Method and system for plesiosynchronous communications with null insertion and removal

ABSTRACT

A serial communications architecture for communicating between hosts and data store devices. The Storage Link architecture is specially adapted to support communications between multiple hosts and storage devices via a switching network, such as a storage area network. The Storage Link architecture specifies various communications techniques that can be combined to reduce the overall cost and increase the overall performance of communications. The Storage Link architecture may provide packet ordering based on packet type, dynamic segmentation of packets, asymmetric packet ordering, packet nesting, variable-sized packet headers, and use of out-of-band symbols to transmit control information as described below in more detail. The Storage Link architecture may also specify encoding techniques to optimize transitions and to ensure DC-balance.

CROSS REFERENCE TO RELATED APPLICATIONS

[0001] This application claims the benefit of U.S. ProvisionalApplication No. 60/252,724 entitled “METHOD AND APPARATUS FOR STORAGEI/O WITH FULL-DUPLEX ONE-TIME BLOCK I/O TRANSFER AND ADAPTIVE PAYLOADSIZING,” filed Nov. 22, 2000, and is related to U.S. patent applicationSer. No. ______ entitled “METHOD AND SYSTEM FOR PLESIOSYNCHRONOUSCOMMUNICATIONS WITH NULL INSERTION AND REMOVAL” (Attorney Docket No.371798002US); U.S. patent application Ser. No. ______, entitled “METHODAND SYSTEM FOR TRANSITION-CONTROLLED SELECTIVE BLOCK INVERSIONCOMMUNICATIONS” (Attorney Docket No. 371798007US); U.S. patentapplication Ser. No. ______ entitled “COMMUNICATIONS ARCHITECTURE FORSTORAGE-BASED DEVICES” (Attorney Docket No. 371798008US1); U.S. patentapplication Ser. No. ______ entitled “METHOD AND SYSTEM FOR PACKETORDERING BASED ON PACKET TYPE” (Attorney Docket No. 371798013US); U.S.patent application Ser. No. ______ entitled “METHOD AND SYSTEM FOR HOSTHANDLING OF COMMUNICATIONS ERRORS” (Attorney Docket No. 371798014US);U.S. patent application Ser. No. ______ entitled “METHOD AND SYSTEM FORDYNAMIC SEGMENTATION OF COMMUNICATIONS PACKETS” (Attorney Docket No.371798015US); U.S. patent application Ser. No. ______ entitled “METHODAND SYSTEM FOR ASYMMETRIC PACKET ORDERING BETWEEN COMMUNICATIONSDEVICES” (Attorney Docket No. 371798016US); U.S. patent application Ser.No. ______ entitled “METHOD AND SYSTEM FOR COMMUNICATING CONTROLINFORMATION VIA OUT-OF-BAND SYMBOLS” (Attorney Docket No. 371798017US);U.S. patent application Ser. No. ______ entitled “METHOD AND SYSTEM FORINTEGRATING PACKET TYPE INFORMATION WITH SYNCHRONIZATION SYMBOLS”(Attorney Docket No. 371798018US); U.S. patent application Ser. No.______ entitled “METHOD AND SYSTEM FOR NESTING OF COMMUNICATIONSPACKETS” (Attorney Docket No. 371798019US); U.S. patent application Ser.No. ______ entitled “COMMUNICATIONS ARCHITECTURE FOR MEMORY-BASEDDEVICES” (Attorney Docket No. 371798012US); U.S. patent application Ser.No. ______ entitled “METHOD AND SYSTEM FOR DC-BALANCING AT THE PHYSICALLAYER” (Attorney Docket No. 371798020US); and U.S. patent applicationSer. No. ______ entitled “MULTISECTION MEMORY BANK SYSTEM” (AttorneyDocket No. 371798021 US, which are all hereby incorporated by referencein their entirety.

BACKGROUND

[0002] The described technology relates generally to communicationstechniques and particularly to communications between hosts and datastore devices.

[0003] The speed and capacity of the data store devices, such as diskdrives and memories, have increased significantly over the past severalyears. As a result of their improved performance, these data storedevices are being used in many new applications, such as databaseservers, Web servers, personal video recorders, and digital displays.These applications often require large amounts of data to becommunicated between data store accessing devices (“hosts”) and datastore devices. (Hosts may include computers, CPUs, or any logic foraccessing a data store device.) Moreover, as host speed increases, thespeed in communicating between hosts and data store devices can have asignificant impact on the overall performance of the application. Inparticular, even though the speed of hosts and data store devices hasincreased significantly, the speed of communications between hosts anddata store devices has not increased as significantly, especially forcommunications over long distances (e.g., greater than one meter). Thus,the communications speed presents a bottleneck in many new applications.

[0004] Current communications techniques typically communicate betweenhosts and certain types of data store devices, such as disk drives,using a bus with many parallel lines or using a single serialcommunications link. The Integrated Disk Electronics (“IDE”) bus and theSmall Computer Systems Interface (“SCSI”) bus are examples of bus-basedparallel communications techniques. These communications techniques,however, present many problems. Performance of bus-based communicationstechniques is generally improved by increasing the number of lines inthe bus, which may significantly increase the cost of such techniques.In addition, bus-based communications techniques generally providearbitration so that multiple hosts and data store devices can share thesame bus. The use of arbitration can significantly increase the cost ofsuch a bus. The cost of such bus-based communication techniques isfurther increased because their design needs to address additionalproblems such as cross-talk and clock skew. In particular, as thecommunications speed increases, the solution to cross-talk and clockskew become much more complex.

[0005] Some serial communications techniques have been developed toaddress some of the problems of bus-based communications techniques.Current serial communications techniques, however, have problems oftheir own. Serial AT, Attachment, which is intended to replace IDE, doesnot scale well and only operates in a half duplex mode. Fibre Channel,currently used to support storage area networks (“SANs”), is verygeneric and therefore, not optimized for any particular application. Inparticular, Fibre Channel has a relatively small packet size with alarge header. As a result, use of Fibre Channel often results in anunacceptably large overhead. For example, data transmitted to diskdrives is typically sent in very large blocks (e.g., 216 bytes). WithFibre Channel, such large blocks need to be divided in many (e.g., 32)packets, which results in a high overhead in the amount of redundantheader information and in the redundant processing performed as a packetis routed to its destination. Thus, Fibre Channel may not be appropriatefor many applications.

[0006] Current memory devices, such as SDRAM and RDRAM, are typicallydesigned to be synchronous with the accessing processing unit. The hostsand the memory devices are synchronous in that they share the same clocksignal. These memory devices are typically optimized for access patternsthat are both temporally and spatially related. In particular, thesememory devices are optimized to read and write arrays (or streams) ofdata. There is a setup overhead (e.g., 5 clock cycles) when accessingthe first word of an array in memory, but access of subsequent words inthe array occurs at the synchronized clock rate (e.g., 1 access perclock cycle). Since the access patterns of central processing units andgraphics processors are typically temporally and spatially related, theycan access such memory devices efficiently.

[0007] Existing memory devices that are designed to support accesspatterns with a high temporal and spatial relationship may not beappropriate for uses having access patterns with a lower spatialrelationship. The setup overhead for each access may be too high. Thereare, indeed, many uses for memory devices with access patterns that arenot as spatially or temporally related as those of a central processingunit or a graphics processor. For example, a switch may have a memorydevice in which packets of data received via an input port are storedbefore they are transmitted via an output port. Traditionally, switchesused crossbars to provide the switching function and FIFOs to provide abuffering function. When a memory device is used on a switch in place ofa crossbar, then all the input and output ports need access to thememory device. The accesses by the different ports are, however, notparticularly spatially related. Moreover, when the packet size is small(e.g., 53 bytes in the case of an ATM switch), the spatial relationshipof accesses by a single port may not be significant. Other uses in whichthere may not be a significant spatial relationship of accesses includenetwork processors and caches for storage area networks. In such uses,the data is received from disparate sources at disparate times and maynot be spatially related.

[0008] Many existing memory devices are not particularly suitable formany uses because the memory devices typically allow access by only oneaccessing device at a time and because the memory devices typicallyoperate at different clock rates than the accessing devices. Becausesuch memory devices can only be accessed by one device at a time, theaccessing devices may need to enter a wait state because the memorydevice is busy or a memory controller may need to have a bufferingcomponent. Of course, the use of a wait state may result in unacceptableperformance. Also, the addition of a buffering component may increasecomplexity and cost. In addition, when multiple accessing devices accessthe same memory device through a single bus (e.g., one writing to thememory device and the other reading from the memory device), then allthe devices that access the memory device need to be synchronized withthe memory device. Because the accessing devices may have differentunderlying clock rates, complex and costly logic is needed to supportthe mapping to the bus clock rate.

[0009] Existing communications protocols, such as Fibre Channel, mayhave an unacceptable overhead for communicating with memory devices. Thecommunications from a host to a memory device may occur in relativelyshort blocks (e.g., 32 bytes). Each block needs to be transmitted in aseparate packet with a relatively large header. In some packets, theheader may be larger than the data itself, which can significantlyreduce the overall bandwidth and speed of transmission. More generally,communications between devices typically occurs in a synchronous or anasynchronous mode. In a synchronous mode, the transmitting and receivingdevices use the same clock signal. The transmitting device can send theclock signal to the receiving device either as a separate signal or as asignal that can be derived from the data signals. When the clock is sentas a separate signal, problems arise resulting from the different delaysin the data signals and the clock signal. These delays and resultingproblems are increased as the transmission speed and distance areincreased. It is very difficult and costly to account for these delays.In addition, the receiving device will have an asynchronous clockboundary. That is, a portion of the receiving device will operate at theclock frequency based on the transmitting device's clock frequency(i.e., the transmitter's clock domain) and another portion will operateat the receiving device's local clock frequency (i.e., the receiver'sclock domain). As a result of the asynchronous boundary, the receivingdevice typically needs to buffer control and data signals sent betweenthe clock domains using elastic buffers, which adds to the complexityand cost of the receiving devices. These elastic buffers requiresubstantial space (e.g., chip area), and when a single chip has multiplecommunications ports, the design is complicated because each port needsits own elastic buffer. When the clock is derived from the data signal,the problems of the delay are reduced somewhat, but there are still theproblems associated with an asynchronous clock boundary.

[0010] A plesiosynchronous clocking technique can be used to avoid theneed to transmit a separate clock signal or derive the clock signal fromthe data signal. With plesiosynchronous clocking (also known as“plesiochronous” clocking), the transmitting and receiving devices haveclocks with nominally the same clock frequency. If the clock frequencieswere exactly the same, then transmitting and receiving devices would besynchronized and the receiving device could accurately identify thetransmitted data (in the case of serial transmission). Also, since thereceiving device operates only at its local clock frequency, there is noasynchronous clock boundary. In practice, however, clock frequencies arenot exactly the same but vary, for example, by 100 ppm. The receivingdevice can use techniques as described in U.S. Pat. No. 6,229,859,entitled “System and Method for High-Speed, Synchronized DataCommunication,” which is hereby incorporated by reference, to accountfor clock variations. Those techniques use an oversampling of the databy the receiving device to detect edge boundaries of the transmitteddata. The receiving device can vary the number of bits of data detectedduring an interval to compensate for the variations in frequency.

[0011] It would be desirable to have a communications architecture thatprovides high-performance for applications (e.g., data storage-basedapplications and memory-based applications) at a low cost. Such acommunications architecture would allow for communications techniques tobe tailored to particular applications.

BRIEF DESCRIPTION OF THE DRAWINGS

[0012]FIG. 1 is a block diagram illustrating components of the StorageLink architecture in one embodiment.

[0013]FIG. 2 is a block diagram illustrating components of the transportlayer, link layer, and physical layer in one embodiment.

[0014]FIG. 3 is a block diagram illustrating components of the physicallayer in one embodiment.

[0015]FIG. 4 is a block diagram illustrating the format of a packet inone embodiment.

[0016]FIG. 5 is a block diagram illustrating the organization of a datapacket and a segmented data packet in one embodiment.

[0017]FIG. 6 is a block diagram illustrating a receive packet memorybefore and after merging of a received packet.

[0018]FIG. 7 is a logic diagram illustrating the segmentation of a datapacket by the transport layer in one embodiment.

[0019]FIG. 8 is a logic diagram illustrating the merging of data packetsby the transport layer in one embodiment.

[0020]FIG. 9A is a diagram illustrating the use of a synchronizationprimitive encoded with the packet type in one embodiment.

[0021]FIG. 9B illustrates the identification of synchronization symbolsin one embodiment.

[0022]FIG. 9C is a block diagram illustrating a component of a linklayer for inserting a synchronization symbol into a stream of symbols.

[0023]FIG. 10 is a block diagram illustrating an organization of apacket memory in one embodiment.

[0024]FIG. 11 is a logic diagram illustrating the process of storingpackets in a transmit packet memory in one embodiment.

[0025]FIG. 12 is a logic diagram illustrating the transport layerprocessing of packets stored in the packet memory in one embodiment.

[0026]FIG. 13 is a diagram illustrating the preemption of a packet inone embodiment.

[0027]FIG. 14 is a logic diagram illustrating the processing of packetpreemption by a transmitting communications node in one embodiment.

[0028]FIG. 15 is a logic diagram illustrating the processing of packetpreemption by a receiving communications node in one embodiment.

[0029]FIG. 16 is a block diagram illustrating asymmetric packet orderingin one embodiment.

[0030]FIG. 17 is a diagram illustrating transaction-based asymmetricpacket ordering in one embodiment.

[0031]FIG. 18 is a logic diagram illustrating processing by a switchthat ensures packet ordering within a transaction in one embodiment.

[0032]FIG. 19A is a block diagram illustrating error handling duringtransmission from a host to a data store device in one embodiment.

[0033]FIG. 19B is a block diagram illustrating the handling of errorsduring the transmission from a data store device to a host in oneembodiment.

[0034]FIG. 19C is a logic diagram illustrating the processing of acommunications node detecting an error in one embodiment.

[0035]FIG. 20 illustrates an alternate bit inversion encoding techniquein one embodiment.

[0036]FIG. 21A is a diagram illustrating the selective block inversionencoding technique in one embodiment.

[0037]FIG. 21B is a block diagram illustrating selective block inversionwith a programmable block size in one embodiment.

[0038]FIG. 21C is a block diagram illustrating transition inversion andpolarity inversion in one embodiment.

[0039]FIG. 22 is a diagram illustrating the format of a primitive in oneembodiment.

[0040]FIG. 23 is a logic diagram illustrating transition control andselective block inversion encoding in one embodiment.

[0041]FIG. 24 is a logic diagram illustrating transition controloptimization in one embodiment.

[0042]FIG. 25 is a logic diagram illustrating block inversion in oneembodiment.

[0043]FIG. 26 is a logic diagram illustrating transition control andselective block inversion decoding in one embodiment.

[0044]FIG. 27 is a logic diagram illustrating the undoing of blockinversion in one embodiment.

[0045]FIG. 28 is a logic diagram illustrating the undoing of transitioncontrol optimization in one embodiment.

[0046]FIG. 29 is a logic diagram illustrating the processing of aprimitive in one embodiment.

[0047]FIG. 30 is a block diagram illustrating the overall architectureof a multiport memory device in one embodiment.

[0048]FIG. 31 is a block diagram illustrating the use of a single phaselock loop for the physical layer of a multiport memory device in oneembodiment.

[0049]FIG. 32 is a block diagram illustrating contents of a bank cachein one embodiment.

[0050]FIG. 33 is a logic diagram illustrating processing of the accesslayer when it receives data from a host in one embodiment.

[0051]FIG. 34 is a logic diagram illustrating the processing of a writecommand by the access layer in one embodiment.

[0052]FIG. 35 is a logic diagram illustrating processing of the accesslayer when it is to provide data to the transport layer in oneembodiment.

[0053]FIG. 36 is a block diagram of a multisection memory bank in oneembodiment.

[0054]FIG. 37A is a block diagram illustrating a variable strength linedriver in one embodiment.

[0055]FIG. 37B is a block diagram illustrating the circuit of avariable-strength line driver in one embodiment.

[0056]FIG. 38A is a block diagram of a plesiosynchronous communicationssystem in one embodiment.

[0057]FIG. 38B is a block diagram illustrating a switch supportingmultiple data store devices in one embodiment.

[0058]FIG. 39A is a block diagram illustrating a prior art synchronousclocking system.

[0059]FIG. 39B is a block diagram illustrating resolution of anasynchronous clock boundary in the physical layer in one embodiment.

[0060]FIG. 40 is a block diagram illustrating NULL insertion anddeletion at the physical layer in one embodiment.

[0061]FIG. 41 is a block diagram illustrating the frame aligner in oneembodiment.

[0062]FIGS. 42A, 42B, and 42C illustrate the loading of the bit bufferin one embodiment.

[0063]FIG. 43 is a diagram illustrating the detection of asynchronization primitive by the sync and null detector in oneembodiment.

[0064]FIG. 44 is a diagram illustrating the portion of the bit bufferthat contains the next symbol to be extracted in one embodiment.

[0065]FIG. 45 is a state diagram of the symbol pointer tracker in oneembodiment.

[0066]FIG. 46 is a diagram illustrating movement of the start-of-symbolpointer due to overrun or underrun in one embodiment.

[0067]FIGS. 47A and 47B are diagrams illustrating NULL detection whilethe start-of-symbol pointer is not within the overrun or underrun extendregions in one embodiment.

[0068]FIGS. 48A and 48B are diagrams illustrating NULL detection whilethe start-of-symbol pointer is in the underrun extend region.

[0069]FIGS. 49A and 49B are diagrams illustrating NULL detection whilethe start-of-symbol pointer is in the overrun extend region.

DETAILED DESCRIPTION

[0070] A serial communications architecture for communicating betweenhosts and data store devices is provided. Aspects of the serialcommunications architecture, referred to as “Storage Link,” arespecially adapted for communications with storage devices, such as diskdrives. Other aspects of the serial communications architecture,referred to as “Memory Link,” are specially adapted for communicationswith memory devices, such as RAM devices. In addition, Storage Link andMemory Link may use an improved plesiosynchronous technique whenreceiving data transmitted via a serial communications link.

[0071] In one embodiment, the Storage Link architecture is speciallyadapted to support communications between multiple hosts and storagedevices via a switching network, such as a storage area network. TheStorage Link architecture specifies various communications techniquesthat can be combined to reduce the overall cost and increase the overallperformance of communications. The Storage Link architecture defines ahierarchy of transport, link, and physical layers such that each layerin the hierarchy is aware of and can take advantage of characteristicsof lower layers in the hierarchy. For example, the transport layer,which is responsible for handling packets, may receive signals from thelink layer that indicate when an end of packet is received. Traditionalcommunications architectures prohibit a lower layer from being aware ofthe functions of a higher layer. The Storage Link architecture mayprovide packet ordering based on packet type, dynamic segmentation ofpackets, asymmetric packet ordering, packet nesting, variable-sizedpacket headers, and use of out-of-band symbols to transmit controlinformation as described below in more detail. The Storage Linkarchitecture may also specify encoding techniques to optimizetransitions and to ensure DC-balance. The Storage Link architecture mayalso use the NULL insertion and removal techniques withplesiosynchronous clocking as described below in detail. The StorageLink architecture thus provides an improved way to access storagedevices that reduces cost and increases communications speed.

[0072] In one embodiment, the Memory Link architecture also providesvarious communications techniques that can be combined to reduce theoverall cost and increase the overall performance of communicationsbetween a host and a memory device. Like the Storage Link architecture,the Memory Link architecture uses a hierarchy of layers in which eachlayer can take advantage of the characteristics of a lower layer in thehierarchy. The Memory Link architecture in one embodiment provides amultiport memory device for serial communications. Each port may use aplesiosynchronous technique when receiving data and may share the samephase lock loop to control sampling of the received data. The MemoryLink architecture may also use a physical layer D-C balancing techniquethat provides an additional driving of a communications link to offsetthe running disparity. The Memory Link architecture may also use animproved plesiosynchronous clock technique that inserts and removesspecial symbols from the received symbols to compensate for variationsin clock frequencies between a transmitting and receiving device.Various combinations of the Memory Link architecture allow for the costof designing, developing, and manufacturing memory-based systems to bereduced and the speed of memory access to be increased.

[0073] In one embodiment, the plesiosynchronous clocking technique usesthe insertion and removal of symbols by the physical layer of areceiving communications node to compensate for variations in clockfrequency between the transmitter and receiver. The receiver maintains asmall buffer (e.g., 27 bits of the received data) at the physical layer.When the receiver detects a synchronization symbol, it initializes astart-of-symbol pointer in the buffer. As subsequent bits of data arereceived, the start-of-symbol pointer is adjusted to account for thevariation in clock frequencies of the transmitter and the receiver. Inparticular, if the transmitter's clock frequency is faster (“an overruncondition”), then the pointer is progressively moved backward in thebuffer (i.e., toward the last received bits). Analogously, if thetransmitter's clock frequency is slower (“an underrun condition”), thenthe pointer is progressively moved forwards in the buffer (i.e., towardthe first received bits). When the receiver detects a special symbol ofa certain type (e.g., a NULL symbol) and the pointer has moved to nearthe beginning or end of the buffer, then the receiver removes thespecial symbol from the buffer to account for the overrun condition orinserts a special symbol into the buffer to account for the underruncondition. In this way, the receiver stays synchronized with thetransmitter even though their clock frequencies vary. Also, asynchronousclock boundaries are avoided in the receivers.

[0074] In the following, aspects of the serial communicationsarchitecture are described using block diagrams and logic diagrams. Oneskilled in the art will appreciate that the serial communicationsarchitecture can be implemented using different combinations of logiccircuits and/or firmware. In particular, the logic diagrams illustrateprocessing that may be performed in parallel using duplicate logiccircuits (e.g., one for each communications link) or may be performed inserial using a single logic circuit. The particular logic designs can betailored to meet the cost and performance objectives of theimplementation of the serial communications architecture. One skilled inthe art will be able to readily design logic circuits based on thefollowing descriptions.

[0075] 1. Storage Link Architecture

[0076]FIG. 1 is a block diagram illustrating components of the StorageLink architecture in one embodiment. The hosts 110 and data storedevices 120 are interconnected to switching network 130 via serialcommunications links 140. The hosts may include computer systems thataccess the data store devices. The data store devices may includestorage area network devices (e.g., disk drives), high-speed memorydevices, and other devices for storing data. The Storage Linkarchitecture, however, is designed to support block-oriented data storedevices, such as disk drives. The switching network may include multipleswitches that are interconnected so that communications paths,especially between hosts and data store devices, can be established.

[0077] Each host, data store device, and switch is a communications nodethat includes one or more communications interfaces 115, 125, and 135with a transport layer, a link layer, and a physical layer. The hostsand data store devices have an upper layer that communicate to eachother via transactions. When transmitting data, the upper layer providesthe data for the transaction to the transport layer. The transport layerreceives the data of the transaction and generates packets (i.e.,packetizes the data) for transmission. The transport layer then provideseach code (e.g., each byte) of the packets to the link layer fortransmission to the destination. When receiving data, the transportlayer receives the codes of transaction from the link layer andidentifies the packets. The transport layer then combines the data ofthe packets of the transaction (i.e., depacketizes the data) andprovides the data of the transaction to the upper layer. Each host anddata store device may include an application upper layer that providesthe data of a transaction to the transport layer to be transmitted to adestination and receives the data of a transaction from the transportlayer that was transmitted by a source. Each packet includes a headersection and a payload section. The header section identifies the sourceand destination and a packet type (e.g., data or command). Each switchmay include a communications interface for each port of the switch. Eachport may be connected to a serial communications link comprising areceive and transmit link. The switches and their interconnectingcommunications links form the switching network. The transport layer ofa switch may not packetize and depacketize the data, rather it maydirect packets received via one port to be transmitted via another portto affect the routing of the packets from the source to the destinationas indicated in the header section.

[0078] The link layer encodes the codes to be transmitted via itscommunications link into symbols and decodes symbols received via itscommunications link into codes. The link layer receives codes to betransmitted from the transport layer, encodes those codes into symbols,and provides those symbols to the physical layer. The link layer alsoreceives symbols from the physical layer, decodes those symbols intocodes, and provides those codes to the transport layer. The encoding mayinclude mapping the codes to symbols to optimize bit transitions and toensure DC-balance. The physical layer receives the symbols from the linklayer, serializes the symbols, and transmits the serialized symbols viathe communications link. The physical layer receives serialized symbolsvia the communications link, deserializes the symbols, and provides thesymbols to the link layer.

[0079]FIG. 2 is a block diagram illustrating components of the transportlayer, link layer, and physical layer in one embodiment. Each layerincludes transmit components 201 and receive components 202. Thetransport layer includes transmit components, such as a transmit memory211, a packetizer 212, and a transmit controller 213, and receivecomponents, such as a receive memory 215, a depacketizer 216, and areceive controller 217. The data of the transaction to be transmitted isstored in the transmit memory. The transmit controller controls thepacketizer to retrieve data of a transaction from the transmit memory,packetize the data, and provide each code of the packets to the linklayer. The receive controller controls the depacketizer to receive codesprovided by the link layer, identify the packets, and store the codes aspackets in the receive memory.

[0080] The link layer includes transmit components, such as an encoder221, a data selector 222, and a control symbol generator 223, andreceive components, such as a decoder 225 and a control symbolidentifier 226. The link layer may be responsible for generating andtransmitting and for receiving and identifying, control symbols,synchronization symbols, timing symbols, and so on, as described belowin more detail. In one embodiment, however, the physical layer isresponsible for detecting control symbols and synchronization symbols.The transport layer may indicate to the link layer when to transmitcontrol symbols, and the link layer may indicate to the transport layerwhen control symbols are received. For example, the transmit layer maynotify the link layer when the receive memory is full. In such a case,the link layer may transmit a control symbol (e.g., XOFF) notifying theother end of the communications link not to transmit any more data. Whensuch transmitted control symbol is received, the link layer of the otherend may indicate to its transport layer to stop transmitting data. Thedata selector selects symbols encoded from codes provided by thetransport layer or symbols generated by the control symbol generator.The data selector effectively inserts control symbols andsynchronization symbols into the sequence of symbols that are to betransmitted. Various possible encoding techniques for the codes aredescribed below in detail. The data decoder receives symbols from thephysical layer and decodes them into codes that are provided to thetransport layer. The control symbol identifier identifies controlsymbols and signals the transport layer as appropriate.

[0081] The physical layer includes a transmitter 231 and receiver 235.The transmitter serializes the symbols provided by the link layer andtransmits the serialized symbols onto the communications link. Thereceiver receives the serialized symbols via the communications link,deserializes the symbols, and provides the symbols to the link layer.

[0082] The transmit components and the receive components can transmitand receive packets in full duplex mode. That is, a packet received bythe transport layer can be transmitted by the transmit componentssimultaneously with a packet being received by the receive components.As discussed below in detail, the encoding techniques enabletransmitting of control symbols by the link layer for link control(e.g., flow control) even while packets are being transmitted in fullduplex mode. That is, link control symbols can be inserted into asequence of symbols generated from codes provided by the transportlayer. Thus, a communications link can be simultaneously transmittingand receiving data symbols, and control symbols can be inserted for linkcontrol. Prior transmission techniques typically transmitted inhalf-duplex mode because the receiving link was reserved fortransmission of control information. Thus, the Storage Link architecturehas a significant bandwidth advantage because of its full-duplex mode.

[0083]FIG. 3 is a block diagram illustrating components of the physicallayer in one embodiment. The physical layer uses a plesiosynchronoustiming mechanism to control the transmission and reception of thesymbols. A plesiosynchronous-based physical layer is described in U.S.Pat. No. 6,229,859, entitled “System and Method for High-SpeedSynchronized Data Communication,” which is hereby incorporated byreference.

[0084] Improvements to the plesiosynchronous timing mechanism aredescribed below in detail. For example, one improvement is the insertionand removal of NULL symbols, which renders the use of elastic buffersunnecessary. Another improvement is the sharing of a single phase lockloop by multiple ports, which avoids the complexity and cost of havingone phase lock loop for each port. The physical layer 230 includes atransmitter 231, a receiver 235, a phase lock loop 301, and a clock 302.The phase lock loop provides a timing signal to the transmitter forserial transmission of the symbols and provides multiple timing signalswith different phases to the receiver for receiving the seriallytransmitted symbols. One skilled in the art will appreciate that a delaylock loop, or more generally any multiphase clock generating device, canbe used in place of the phase lock loop. The receiver may include anoversampler 303, a digital phase lock loop 304, and a frame aligner 305.The digital phase lock loop selects the best sample by a process knownas majority voting. Majority voting counts transitions from oneoversampled bit to another. In one embodiment, each bit is oversampledthree times and the position with the highest number of transitions isselected. The digital phase lock loop thus identifies the bit values ofthe received symbols, and the frame aligner aligns the bits into symbolsand provides the deserialized symbols.

[0085]FIG. 4 is a block diagram illustrating the format of a packet inone embodiment. The packet 400 includes a header section 410 and apayload section 420. The header section includes reserved fields 411 and412 (e.g., 16 bits each), a destination address field 413 (e.g., 16bits), a source address field 414 (e.g., 16 bits), a type field 415(e.g., 8 bits), a protocol field 416 (e.g., 8 bits), a class specificcontrol field (“CSF”) 417 (e.g., 8 bits), and a transaction (or queue)index field 418 (e.g., 8 bits). One skilled in the art will appreciatethat the actual size of the fields can vary depending on the desiredperformance characteristics. For example, the destination and sourceaddress fields can be limited to 4 bits when there are at most 16 hostsor data store devices. The class-specific control field contains thecontrol field of the interface class (e.g., SCSI, ATA), which may beunused in some interface classes. The protocol field identifies theappropriate protocol for the upper layer and identifies the protocol ofthe storage media (e.g., SCSI, ATA, or Storage Link). The destinationaddress field identifies the host or data store device to which thepacket is to be transmitted. The source address field identifies thehost or data store device that originated the transmission of thepacket. In one embodiment, the reserved fields may be used inconjunction with the source address field and the destination addressfield to permit 32-bit addresses. The type field identifies the type ofthe packet. The types of packets may include a data packet, a commandpacket, a status packet, and a message packet. The command packet,status packet, and message packet are “control” packets. Control packetsare generally much smaller than data packets. The type field furtherspecifies the direction of the packet transmission (i.e., whether it isbeing transmitted from a host to a data store device, or vice versa) andspecifies whether the packet is intended for the physical layer, linklayer, transport layer, or application layer. For example, anapplication layer packet may be a command packet sent from a host deviceto a data store device requesting that data be retrieved from the datastore device. The payload section contains type-specific information.For example, the payload section of a command packet for retrieving datafrom a data store device may include the starting address of the data tobe retrieved and the number of bytes of data to be retrieved. Thepayload section of a data packet transmitted in response to receiving acommand packet may contain the retrieved data in the payload section.The control field contains type-specific information that may be used tocontrol the data store device. The transaction index field identifies aseries of packets that correspond to the same transaction. For example,a transaction may be the writing of data to a data store device. A hostmay transmit the data to be written in multiple packets that form asingle transaction. Each packet within the transaction contains the sametransaction index so that the data store device can identify that thepackets belong to the same transaction. Although not shown, a packet mayalso include a cyclic redundancy check (CRC) field after the payloadsection.

[0086] In various embodiments, the size of the header portion can varybased on the number of hosts and data store devices that areaddressable. For example, if a single host is directly connected to asingle data store device, then a small header including the type field,control field, protocol field, a class specification control field andtransaction index field would be sufficient. The destination and sourceaddress fields and reserved fields would not be needed. When there areeither multiple hosts or multiple data store devices, then thedestination and source address fields would be included in the header.If more hosts and data store devices are addressable than can beaddressed by 16 bits, then the reserved fields can be included in theheader. Thus, the size of the header can vary depending on the number ofpossible sources and destinations. In one embodiment, the transportlayer may be configurable to include a small, medium, or large headerdepending on the number of possible sources and destinations.

[0087] A packet that is transmitted on a serial communications linktypically comprises in-band symbols with certain desirablecharacteristics, such as with its number of bit transitions optimized,as described below in more detail. The physical layer continuallytransmits symbols onto the serial communications link. If a packet isnot available to be transmitted, then the link layer provides IDLEsymbols to the physical layer for transmission. In one embodiment, anIDLE symbol is an out-of-band symbol, which is a symbol that does notnormally appear within a packet because it may not have the samedesirable characteristic as an in-band symbol (e.g., bit transitions notoptimized). A receiving link layer identifies the start of a packet by atransition from receiving IDLE symbols to receiving in-band symbols.Similarly, a receiving link layer identifies the end of a packet by atransition from receiving in-band symbols to receiving IDLE symbols. Inone embodiment, a IDLE symbol is actually a sequence of two out-of-bandsymbols for reasons described below in detail. More generally, asequence of two out-of-band symbols is a primitive that is used toexchange control information (e.g., XON and XOFF) between link layers asdescribed below in detail. Because control information is encoded asout-of-band symbols, the control symbols can be inserted into a sequenceof in-band symbols and readily detected as control symbols whenreceived. The control symbols can be interspersed with the transmissionof symbols of a packet so that one end of a communications link canprovide link control information to the other end even while packets arebeing transmitted in both directions (i.e., full-duplex mode)

[0088] 1.1 Packet Segmentation

[0089] In one embodiment, a packet (e.g., long packets, such as datapackets) may be dynamically divided or segmented into multiple packetsby a communications node (e.g., a host, data store device, or switch) asthe packet is being transmitted from a source to a destination. Thedynamic segmentation of packets allows for the efficient transmission oflarge packets via a single path when the switching network is notcongested and allows for the efficient transmission of segmented packetson different paths when congestion prevents use of a single path. Inaddition, dynamic segmentation may avoid the resending of portions ofthe payload when segmenting a packet.

[0090] A packet may be segmented by a communications node when, forexample, the next communications node in the path indicates that itcannot receive any more symbols of a packet. The transmittingcommunications node may receive an XOFF signal from the receivingcommunications node (e.g., because of a buffer overflow at the receivingcommunications node) in the middle of transmitting a packet. Thetransmitting communications node stops transmitting the packet until anXON signal is received from the receiving communications node or unlessit can start transmitting the remainder of the packet on a differentcommunications link (i.e., through a different port). In either case,the transmitting communications node transmits a new header with theremainder of the packet, which results in the segmentation of the packetinto a first packet with a first header and a second packet with asecond header. The packet segmentation is performed by the transportlayer based on signals provided by the link layer (e.g., in response toreceiving an XOFF signal).

[0091]FIG. 5 is a block diagram illustrating the organization of a datapacket and a segmented data packet in one embodiment. The payloadsection of a data packet may include a target address and blocks of datato be stored at the target address. For example, if the data storedevice is a disk drive, then the target address may be a sector addressof the disk drive and the blocks of data may each correspond to a sectorof data. In one embodiment, the blocks have a fixed size (e.g., thesector size). Packet 510 includes a header section 500 and a payloadsection 511 that includes an address 500 a and blocks 501-506. When acommunications node is transmitting packet 510, it may transmit theheader section and address 500 a and blocks 501-503 of the payloadsection. During transmission of block 504, the transmittingcommunications node may receive an XOFF signal from the receivingcommunications node. (The receiving communications node may transmit anXOFF control primitive.) The transmitting communications node completesthe transmission of block 504 and then stops transmitting the packet onthat serial communications link. (The link layer then startstransmitting an IDLE symbol on the communications link.) When thetransmitting communications node eventually receives an XON signal fromthe receiving communications node, it transmits a new header section 531plus a payload section indicating a new address 531 a and blocks 505 and506 to complete the transmitting of the packet. The new address may berepresented by an offset that is added to a start address to generatethe new address. The new header may contain the same header informationas the previously transmitted header, but the address (e.g., as anoffset) of the payload section is updated to reflect the startingaddress at which the remaining blocks of the packet are to be stored.For example, if packet 510 is segmented after block 504 and the startingaddress for that packet is address 211, then the starting address forthe packet containing blocks 505 and 506 is 215 (i.e., 211+4). Packet510 is thus segmented into packets 520 and 530. When a packet issegmented, the transport layer may route the remaining packet on adifferent communications link to the destination. The use of differentcommunications links helps to optimize the use of available bandwidth.

[0092] In addition to segmenting packets, the transport layer may mergepackets that were previously segmented. For example, if packet 520 iscurrently stored in a receive memory of a switch awaiting transmissionand packet 530 is received at the switch, then the transport layer mayrecognize that the packets are part of the same transaction (e.g., basedon source addresses, destination address, and transaction index) andthat their blocks are contiguous as indicated by their target addresses.The transport layer can merge packets 520 and 530 to form packet 510.One skilled in the art will appreciate that packets 520 and 530 can bemerged regardless of the order in which they were received at theswitch. Also, one skilled in the art will appreciate that if a switch isin the process of transmitting packet 520 and packet 530 is received,the packets can be merged by transmitting blocks 505 and 506 immediatelyafter block 504. One skilled in the art will also appreciate that atransport layer can merge packets stored in a transmit memory. In thisway, packets received via different ports of a switch can be merged. Themerging of packets allows data of the same transaction to be moreefficiently transmitted when traffic on the switch network allows.

[0093]FIG. 6 is a block diagram illustrating a receive packet memorybefore and after merging of a received packet. Packet 600 formingtransaction T1 initially contains 8 blocks. During transmission, it issegmented into packets 614, 630, and 612. Packets 614 and 612 arecurrently buffered at a switch which then receives packet 630. Becausethe switch has all three packets, they can be merged into a singlepacket. Before the merge, packet memory 610 contains 4 packets. Packet611 is part of transaction T3 and contains 5 blocks of data to be storedstarting at address 25. Packet 612 is part of transaction T1 andcontains 3 blocks of data to be stored starting at address 4. Packet 613is part of transaction T2 and contains 10 blocks of data to be storedstarting at address 50. Packet 614 is part of transaction T1 andcontains 2 blocks of data to be stored starting at address 0. Packets612 and 614 are both part of transaction T1. These packets, however, arenot contiguous packets within transaction T1 and thus cannot be mergedtogether. Packet 630 represents a packet just received that is part oftransaction T1 and contains 2 blocks to be stored starting at address 2.Packet 630 is contiguous to both packets 614 and 612. That is, packet630 can be merged onto the end of packet 614 and packet 612 can bemerged onto the end of packet 630. Packet memory 620 contains thepackets after the merge. Packets 611 and 613 have been left unchanged.Packet 614, however, has been modified to add the blocks from packet 630and from packet 612. Packet 614 indicates that is the part oftransaction T1 and contains 7 blocks of data to be stored starting ataddress 0. Packet 612 has been deleted from the packet memory because ithas been merged with packet 614, and packet 630 has not been separatelyadded to the packet memory because it has been merged with packet 614.

[0094]FIG. 7 is a logic diagram illustrating the segmentation of a datapacket by the transport layer in one embodiment. In block 701, acomponent of the transport layer waits for a block of data. In decisionblock 702, if an XOFF signal has been received, then the component waitsfor an XON signal in block 703, else the component continues at block704. In block 704, the component sends the header to the link layer fortransmission. In blocks 705-707, the component transmits successiveblocks of the packet and segments the packet when an XOFF signal isreceived from the link layer. In block 705, the component transmits thenext block of the packet. In decision block 706, if an XOFF signal ispending as indicated by the link layer, then the component continues tosegment the packet at block 708, else the component continues at block707. In decision block 707, if the next block is ready, then thecomponent loops to block 705 to transmit the next block, else thecomponent segments (or ends) the packet at block 708. In block 708, thecomponent transmits the CRC to end the packet and waits for the nextblock in 701.

[0095]FIG. 8 is a logic diagram illustrating the merging of data packetsby the transport layer in one embodiment. In block 801, a component ofthe transport layer receives a data packet from the link layer. Inblocks 802-807, the component checks to see if the data packet can bemerged with a previously received data packet that is stored in thereceive memory. In block 802, the component selects the next data packetin the receive memory. In decision block 803, if all the data packets ofthe receive memory have already been selected, then the componentcontinues at block 808, else the component continues at block 804. Indecision block 804, if the selected and received data packets arecontiguous, then the component continues at block 805, else thecomponent selects the next data packet in the receive memory. In block806, the component adds the blocks of the selected packet to the receivepacket to affect the merging of the blocks. In block 807, the componentremoves the selected packet from the receive memory. The component thenselects the next data packet because there may be another data packetthat can be merged at the beginning or end of the received data packet.In block 808, the component adds the received data packet to the receivememory and then completes.

[0096] 1.2 Synchronization Primitive and Packet Type

[0097] In one embodiment, the serial communications architecture definesmultiple primitives to operate as synchronization signals. Asynchronization signal is typically used by the physical layer to helpensure correct alignment of symbols. A transmitting communications nodeperiodically transmits synchronization primitives so that the receivingcommunications node can properly align with that synchronizationprimitive. When a communications node receives a synchronizationprimitive, it knows that the primitive is correctly aligned on a symbolboundary. The use of multiple synchronization primitives allows forencoding packet type within a synchronization primitive. In particular,the transport layer provides the packet type to the link layer. The linklayer can then transmit a synchronization primitive that indicatespacket type immediately before the packet itself. The receivingcommunications node can take the appropriate action based on the packettype encoded in the synchronization primitive without having to parsethe packet header. Indeed, the packet header may not need to separatelyinclude the packet type. The use of a synchronization primitive withencoded packet type may be well adapted to use in an environment wherethe transmission time of a packet through the switching network may belong relative to the time needed to store the packet at the data storedevice. For example, the time to transmit a packet through the switchingnetwork may be longer than the time needed to store the data of thepacket in a memory-based data store device. The rapid detection ofpacket type through a synchronization primitive, rather than through apacket header, can reduce the processing time of a switch. Also, when asynchronization primitive encodes the packet type, then the link layer,rather than the transport layer, can detect packet type, which canenhance the overall processing speed.

[0098]FIG. 9A is a diagram illustrating the use of a synchronizationprimitive encoded with the packet type in one embodiment. Serial linkcontent diagram 900 indicates the information transmitted on a seriallink over a period of time. Initially, an IDLE primitive is transmittedonto the communications link. (An IDLE primitive may be asynchronization primitive that does not encode any packet type.)Immediately prior to transmitting control packet 902 (e.g., command,status, or message packets), the transport layer of the transmittingcommunications node signals the link layer that a control packet is tobe transmitted next. The link layer then transmits a synchronizationprimitive 901 encoded with a packet type of control. After the packethas been transmitted, the link layer of the transmitting communicationsnode starts transmitting the idle primitive. Immediately before thetransport layer transmits data packet 904, it notifies the link layerthat a data packet is to be transmitted. The link layer transmits asynchronization primitive 903 encoded with a packet type of data. Afterthe data packet is transmitted, the link layer starts transmitting theIDLE primitive. The receiving communications node uses thesynchronization primitive encoded with packet type to align its symbolboundaries and to rapidly identify the packet type. This alignment andidentification can occur in the physical and link layers.

[0099] As described above, the primitives are pairs of out-of-bandsymbols. (In the following, a symbol is assumed to have nine bits.) Inone embodiment, when the in-band symbols are transition maximized, thenthe symbols of the control primitives have three transitions each andthe symbols of the synchronization primitives have two transitions each.When the in-band symbols are transition minimized, then the symbols ofthe control primitives have five transitions each and thesynchronization primitives have one symbol with five transitions andanother symbol with seven transitions.

[0100] Table 1 illustrates control primitives in one embodiment when thein-band symbols are transition maximized. Each control primitivecomprises two symbols in which the first symbol is a polarity inversionof the second symbol. As a result, the disparity (described below) ofeach control primitive is 0. The control primitives such as XON, XOFF,preempt, and continue are selected from this table of controlprimitives. TABLE 1 Control primitives in transition maximization mode0. 000010111-111101000 1. 000011011-111100100 2. 000011101-111100010 3.000100111-111011000 4. 000110011-111001100 5. 000111001-111000110 6.001000111-110111000 7. 001100011-110011100 8. 001110001-110001110 9.010000111-101111000 10. 011000011-100111100 11. 011100001-100011110 12.100001110-011110001 13. 100011100-011100011 14. 100111000-011000111 15.101110000-010001111 16. 110000110-001111001 17. 110001100-001110011 18.110011000-001100111 19. 110110000-001001111 20. 111000010-000111101 21.111000100-000111011 22. 111001000-000110111 23. 111010000-000101111

[0101] Table 2 illustrates control primitives in one embodiment when thein-band symbols are transition minimized. TABLE 2 Control primitives intransition minimization mode 0. 000010101-111101010 1.000100101-111011010 2. 000101001-111010110 3. 000101011-111010100 4.000101101-111010010 5. 000110101-111001010 6. 001000101-110111010 7.001001001-110110110 8. 001001011-110110100 9. 001001101-110110010 10.001010001-110101110 11. 001010011-110101100 12. 001011001-110100110 13.001100101-110011010 14. 001101001-110010110 15. 010000101-101111010 16.010001001-101110110 17. 010001011-101110100 18 010001101-101110010 19.010010001-101101110 20. 010010011-101101100 21. 010011001-101100110 22.010100001-101011110 23. 010100011-101011100 24. 010110001-101001110 25.011000101-100111010 26. 011001001-100110110 27. 011010001-100101110 28.100001010-011110101 29. 100010010-011101101 30. 100010100-011101011 31.100010110-011101001 32. 100011010-011100101 33. 100100010-011011101 34.100100100-011011011 35. 100100110-011011001 36. 100101000-011010111 37.100101100-011010011 38. 100110010-011001101 39. 100110100-011001011 40.101000010-010111101 41. 101000100-010111011 42. 101000110-010111001 43.101001000-010110111 44. 101001100-010110011 45. 101010000-010101111 46.101011000-010100111 47. 101100010-010011101 48. 101100100-010011011 49.101101000-010010111 50. 110001010-001110101 51. 110010010-001101101 52.110010100-001101011 53. 110100010-001011101 54. 110100100-001011011 55.110101000-001010111

[0102] The synchronization primitives are distinct from the controlprimitives. The synchronization primitives are selected to have thecharacteristic that when two synchronization primitives are transmittedsequentially the receiving device can unambiguously identify thesynchronization primitives regardless of what is transmitted before orafter the synchronization primitives. The synchronization primitiveshave a unique number of transitions that is not found in any controlprimitive or in-band symbol. In one embodiment, the synchronizationprimitive can be detected by looking for a sequence of symbols with thatnumber of transitions. In one embodiment, the synchronization primitivesare selected to start with a falling edge (i.e., “10”), a rising edge(i.e., “01”), a high level (i.e., “11”), or a low level (i.e., “00”).The selection of the start of the synchronization primitives is referredto as the “synchronization detection mode.” Table 3 containssynchronization primitives in transition maximization mode that startwith a falling edge and a rising edge. Each symbol in Table 3 containstwo transitions resulting in synchronization primitives with fourtransitions. Table 4 contains synchronization primitives in transitionminimization mode that start with a low level and a high level. Eachsynchronization primitive in Table 4 has 13 transitions. The high-leveland low-level synchronization primitives have five transitions in onesymbol, seven transitions in the other symbol, and one transition fromtheir first to second symbol. TABLE 3 Falling Edge Rising Edge 0100000001-100111111 011000000-011111110 1 100000011-100011111011100000-011111100 2 100000111-100001111 011110000-011111000 3100001111-100000111 011111000-011110000 4 100011111-100000011011111100-011100000 5 100111111-100000001 011111110-011000000

[0103] TABLE 4 High Level Low Level 0 110101000-110101010001010101-001010111 1 110100010-110101010 001010101-001011101 2110001010-110101010 001010101-001110101 3 110100100-110101010001010101-001011011 4 110010100-110101010 001010101-001101011 5110101010-110101000 001010111-001010101 6 110101010-110100010001011101-001010101 7 110101010-110001010 001110101-001010101 8110101010-110100100 001011011-001010101 9 110101010-110010100001101011-001010101

[0104] Table 5 lists the transition distribution of the low and highpolarity symbols of the synchronization primitives. TABLE 5 Total # ofTrans- itions Sync Detect Transition Distribution TransitionDistribution in a Sync Mode in Low Polarity in High Polarity PrimitiveFalling edge one “10” and one “01” one “10” and one “01” 4 (“10”) Risingedge one “01” and one “10” one “01” and one “10” 4 (“01”) High level one“11” and two “00” one “11” and no “00” 13 (“11”) Low level one “00” andno “11” one “00” and two “11” 13 (“00”)

[0105] In one embodiment, the synchronization primitives are detected bybuffering three symbols (e.g., 27 bits). The synchronization primitivesare selected so that any time the buffer has exactly three of thesynchronization detection mode bits (e.g., “10” for falling edge) thatare each 9 bits apart then the bits are at the beginning of asynchronization symbol. The bit buffer contains 27 bits, A[8:0], B[8:0],and C[8:0], in which bit A[8] was received first and bit C[0] wasreceived last. In this example, the content of bits A[6]:B[7] representsynchronization symbol “100000001” and B[6]:C[7] representsynchronization symbol “100111111.” The starting points of thesynchronization symbols (“synchronization markers”) are A[6], B[6], andC[6].

[0106]FIG. 9C is a block diagram illustrating a component of a linklayer for inserting a synchronization symbol into a stream of symbols.The link layer 910 is provided with the bits of the code (“Data”) to betransmitted and with an indication of a control primitive (“CTL-S”) totransmit. The idle inserter 910 of the link layer automatically insertsan IDLE primitive (e.g., synchronization primitive) to ensure that nottoo may data symbols have been sent without sending a synchronizationprimitive. In one embodiment, the transmitter may send an IDLE primitiveafter every block of a packet. If packets are short, the inter-packetgap (“IPG”) is sufficient for clock compensation. That is, the receiverwill receive synchronization primitives often enough to maintain propersynchronization. If a packet is long, however, the link layer canperiodically insert a synchronization primitive to ensure that thereceiver maintains proper synchronization. The multiplexer of the linklayer selects the appropriate symbol to send to the physical layer.

[0107] 1.3 Transmission Order Based on Packet Type

[0108] In one embodiment, a communications node transmits packets ofcertain types before packets of other types regardless of the order inwhich the packets are received or generated by the communications node.For example, a switch may transmit control packets (e.g., commandpackets, status packets, and message packets) before transmitting datapackets. The transmitting of control packets before data packets mayhelp ensure that the control packets are not unnecessarily delayed bydata packets, which may have a tendency to be quite large. For example,a command packet requesting a read from a data store device may betransmitted before a data packet even through the control packet wasreceived after the data packet. The data store device can thus startprocessing the read request without having to wait until the data packetis transmitted. A communications node may store control packets and datapackets in separate areas of the packet memory and only select packetsfrom the data packet area when the control packet area is empty.Alternatively, the communications node may use various algorithms toselect control or data packets to be transmitted. For example, thealgorithm may select a data packet that has been in the packet memoryfor certain amount of time even though there may be a control packet inthe control area. In this way, a sequence of many control packets willnot delay the data packet more than that certain amount of time. Oneskilled in the art will appreciate that there may be multiple packettypes with different transmission orders. For example, a status packetmight be transmitted before a command packet, and a command packet mightbe transmitted before a data packet.

[0109]FIG. 10 is a block diagram illustrating an organization of apacket memory in one embodiment. The transmit packet memory 211 mayinclude control queue 1010 and data queue 1020. The control queuecontains the control packets 1011, and the data queue contains the datapackets 1021. Packets to be transmitted are stored either in the controlqueue or the data queue based on packet type (which may be encoded in asynchronization primitive). When a transmitting communications node isready to transmit the next packet, it retrieves the next packet from thecontrol queue. If the control queue is empty, then the transmittingcommunications node retrieves the next packet from the data queue.

[0110]FIG. 11 is a logic diagram illustrating a process of storingpackets in a transmit memory in one embodiment. Similar logic can beused to store packets in the receive memory of a communications node. Inblock 1101, a component of the transport layer receives the next packetfrom an upper layer. In decision block 1102, if the received packet is acontrol packet, then the component continues at block 1103, else thecomponent continues at block 1104. The component may determine the typeof the packet either from the type field of the packet header or, whenthe packet is received from another communications node, from asynchronization primitive encoded with the packet type. When the linklayer detects a synchronization primitive designating packet type, itsignals the transport layer so that the type of packet can be used whenthe packet is stored in the receive memory, retrieved from the receivememory, stored in the transmit memory, and retrieved from the transmitmemory. In block 1103, the component stores the packet in the controlqueue. In block 1104, the component stores the packet in the data queue.The component then retrieves the next packet.

[0111]FIG. 12 is a logic diagram illustrating the transport layerprocessing of packets stored in the packet memory in one embodiment. Indecision block 1201, if the control queue is empty, then the componentcontinues to block 1203, else the component retrieves a control packetfrom the control queue in block 1202 and continues at block 1205. Indecision block 1203, if the data queue is empty, then the componentagain checks for a packet in the control queue, else the componentretrieves a packet from the data queue in block 1204 and continues atblock 1205. In blocks 1205-1207, the component selects the codes of theretrieved packet. In block 1205, the component selects the next code ofthe retrieved packet. In decision block 1206, if all codes have alreadybeen selected, then the component continues at block 1201 to check forthe next packet, else the component continues at block 1207. In block1207, the component provides the selected code to the link layer andthen continues at block 1205 to select the next code of the packet.

[0112] 1.4 Packet Preemption

[0113] In one embodiment, a communications node may preempt or interruptthe transmission of a packet when a packet with a higher priority is tobe transmitted on the same communications link. Packet preemption helpsensure that data store devices receive control packets as soon aspossible without being slowed by network congestion as a result of datapackets. In particular, the transmission of a data packet, which may belarge, may be preempted so that a control packet, which is typicallysmall, may be transmitted. When a data packet is to be preempted, thetransport layer of the communications node stops providing the codes ofthe data packet to the link layer, signals the link layer to transmit apreempt primitive, and then starts providing the code of the controlpacket to the link layer for transmission as a preempting packet. Whenthe preempting packet has been completely transmitted, the transportlayer of the communications node signals the link layer to transmit acontinue primitive and then resumes providing the remainder of the codesof the data packet to the link layer. The link layer of the receivingcommunications node detects the preempt primitive while it is receivingthe codes of the data packet and signals its transport layer. Thetransport layer then starts storing the subsequent codes as a controlpacket. When the link layer detects the continue primitive, it signalsthe transport layer, which continues receiving the preempted datapacket. The preempt and continue primitives, which are controlprimitives, serve to delimit transmission of a preempting packet. Oneskilled in the art will appreciate that packet preemption could benested to any level. For example, a status packet might preempt acommand packet that already preempted a data packet.

[0114]FIG. 13 is a diagram illustrating the preemption of a packet inone embodiment. Serial link content diagram 1300 illustrates a datapacket that is preempted by a control packet. Initially, thetransmitting communications node transmits the first portion of datapacket 1301. When the packet is to be preempted, the communications nodethen transmits the preempt primitive 1302. After transmitting thepreempt primitive, the transmitting communications node transmits thepreempting control packet 1303. After the control packet is transmitted,the communications node transmits the continue primitive 1304, and thenresumes transmitting the remainder of the preempted data packet 1305.

[0115]FIG. 14 is a logic diagram illustrating the processing of packetpreemption by a transmitting communications node in one embodiment. Inblock 1401, the component of the transport layer retrieves the nextpacket from the transmit memory. In block 1402, the component retrievesthe next code of the retrieved packet. In decision block 1403, if allthe codes have already been retrieved from the packet, then thecomponent continues at block 1401 to retrieve the next packet, else thecomponent continues at block 1404. In block 1404, the component providesthe retrieved code to the link layer. In decision block 1405, if apreempt signal has been raised (e.g., as a result of being provided acontrol packet by an upper layer), then the component continues at block1406, else the component continues at block 1402 to retrieve the nextcode of the packet. In block 1406, the component notifies the link layerto transmit a preempt primitive. In blocks 1407-1409, the componentprovides each code of the preempting packet to the link layer. In block1407, the component selects the next code of the preempting packet. Indecision block 1408, if all the codes of the preempting packet havealready been selected, then the component continues at block 1410, elsethe component continues at block 1409. In block 1409, the componentprovides the selected code to the link layer and then continues at block1407 to select the next code of the preempting packet. In block 1410,the component signals the link layer to transmit a continue primitiveand then continues processing the preempted packet at block 1402.

[0116]FIG. 15 is a logic diagram illustrating processing of packetpreemption by a receiving communications node in one embodiment. Inblock 1501, the component of the transport layer retrieves the next codefrom the link layer. In decision block 1502, if the link layer signalsthat a preempt primitive has been received, then the component continuesat block 1504, else the component continues at block 1503. In block1503, the component adds the code to the current packet and retrievesthe next code in block 1501. In block 1504, the component saves currentpacket information so that it can continue storing codes in the packetafter the preemption is complete. In block 1505, the component processesthe preempting packet. The processing includes receiving each code andstoring the codes in a preempting packet until the link layer indicatesthat it received a continue primitive. In block 1505, the componentrestores the saved packet information so that subsequent received codescan be stored in the preempted packet. The component then selects thenext code of the preempted packet in block 1501.

[0117] 1.5 Asymmetric Packet Ordering

[0118] In one embodiment, the serial communications architecture mayguarantee that packets (in particular packets of the same transaction)transmitted from a host to a data store device are received at the datastore device in the order in which they were transmitted from the host.In contrast, the serial communications architecture might not guaranteethat packets transmitted from a data store device to a host are receivedat the host in the same order that they were transmitted from the datastore device. The guarantee of order in one direction, but not in theother direction, is referred to as asymmetric packet ordering. Becausethe packets are received at the data store device in order, the datastore device does not need the hardware or software necessary to reorderpackets that are received out-of-order. As a result, the overall cost ofthe data store device can be reduced. The guarantee of order may beprovided by requiring that all packets of the same transactiontransmitted from a host to a data store device travel on the same path.In contrast, packets of the same transaction transmitted from a datastore device to a host may each travel on different paths. Because thepackets transmitted from a host to a data store device travel along thesame path, the actual time to transmit the packets may take longer thanif alternate paths could be used. For example, one communications linkin the path may become congested with no alternative but to wait for thecongestion to end, whereas if an alternate path was used, the remainingpackets could be routed to avoid the congestion; thus, the packets wouldarrive sooner, but possibly out-of-order. In one embodiment, the serialcommunications architecture may only guarantee the order of packetstransmitted from a host to a data store device within each transaction.Packets within a transaction have a sequential order. The guarantee oforder within a transaction may be provided by requiring that all thepackets of each transaction to be transmitted from a host to a datastore device along the same path. The packets of different transactionscan, however, be transmitted along different paths. In this way, apacket within a later transmitted transaction may arrive at the datastore device before a packet of an earlier transmitted transaction.

[0119] A host may be a computer system running application programs, andthe data store device may be a disk drive. In such an environment, thespeed at which data is transmitted to a disk drive may not be asimportant as the speed at which the data is transmitted to the computersystem. In particular, when an application program writes data to a diskdrive, the data is normally cached by the computer system beforetransmission to the disk drive so that the application program cancontinue its processing without having to suspend its execution. Incontrast, when an application program reads data from a disk drive, theapplication normally suspends its execution until the data is received.Thus, the speed of transmitting data from the disk drive is moreimportant than the speed of transmitting data to a disk drive because itcan reduce the time a program is suspended. The asymmetric packetordering takes advantage of this variation in the need for speed.

[0120]FIG. 16 is a block diagram illustrating asymmetric packet orderingin one embodiment. Packets of the same transaction, transmitted from ahost 1610 to a data store device 1620 all travel along the same path:switch 1631, switch 1634, and switch 1636. Since the data packets areprocessed in a first-in-first-out manner by each switch, packet orderingis preserved in transit. In contrast, packets of the same transactiontransmitted from a data store device 1620 to a host 1610 may travelalong two different paths: (1) a path including switch 1636, switch1633, and switch 1631 and (2) path including switch 1636, switch 1635,switch 1632, and switch 1631. Packets traveling on different paths mayarrive out-of-order at the host.

[0121]FIG. 17 is a diagram illustrating transaction-based asymmetricpacket ordering in one embodiment. Illustration 1701 illustratespreserving packet ordering within a transaction transmitted from a hostto a data store device. In this example, transaction 1 includes packets1 and 2 and transaction 2 includes packets 1, 2, 3, and 4. Transaction 1is transmitted from the host before transaction 2. The data store devicereceived the packets of each transaction in order within theirtransaction, but some of the packets of transaction 2 arrived before thesome of the packets of transaction 1. For example, packet 1 oftransaction 2 arrived before packet 1 of transaction 1, but after packet1 of transaction 2. Illustration 1702 illustrates transmitting withoutpreserving packet ordering within a transaction. In this example,transactions 1 and 2 are transmitted from the data store device to thehost. The host receives the packets of the transactions out of order.For example, packet 2 of transaction 2 arrived before packet 1 oftransaction 2.

[0122]FIG. 18 is a logic diagram illustrating processing by a switchthat ensures packet ordering within a transaction in one embodiment. Inblock 1801, a component of the switching layer retrieves the next datapacket to be routed. In decision block 1802, if another packet of thesame transaction has been transmitted by the switch, then the componentidentifies the same communications link in block 1803, else thecomponent identifies a communications link for the first packet of thattransaction.

[0123] 1.6 Host Error Handling

[0124] In one embodiment, the communications nodes detect errors but donot attempt to correct or handle the errors. Rather, communicationsnodes forward error messages to the communications node that initiatedthe transaction associated with the error. For example, a host mayrequest a data store device, such as a disk drive, to provide datastored at the data store device. If an error occurs (e.g., a CRC error)while the data is being transmitted from the data store device to thehost, the communications node (e.g., a switch) that detects the errortransmits an error message to the initiating host. The initiating hostcan then handle to the error as appropriate. For example, the initiatinghost may retransmit the request to retrieve the data from the data storedevice. Because the communications node (e.g., data store device orswitch) does not attempt to handle errors, the complexity of thesecommunications nodes is reduced. Although the handling of the error bythe initiating host may be slow, the slowness may be acceptable becauseerrors are expected to be relatively uncommon.

[0125]FIG. 19A is a block diagram illustrating the handling of errorsduring transmission from a host to a data store device in oneembodiment. The host 1901 transmits a packet 1904 to switch 1902. Theswitch detects an error, transmits an error message packet 1905 to thehost reporting the error, and suppresses further transmission of thepacket to the data store device 1903. When the host receives the errormessage packet, it forwards an error message 1906 to the upper layer forprocessing. Similarly, when the data store device detects an error inthe transmission of packet 1907 from the switch, it transmits an errormessage packet 1908 reporting the error to the switch, which is thentransmitted as an error message packet 1909 to the host, and the errormessage 1910 is eventually reported to the upper layer of the host.

[0126]FIG. 19B is a block diagram illustrating the handling of errorsduring the transmission from a data store device to a host in oneembodiment. The switch 1902 detects an error in the transmission ofpacket 1911 from the data store device 1903. Rather than transmitting anerror message packet 1912 to the data store device, the switch transmitsan error message packet 1913 to the host 1901 that initiated thetransaction of the packet 1911 for which the error was detected. Uponreceiving the error message packet, the host forwards an error message1914 to an upper layer for handling. Similarly, when the host detects anerror in transmission of packet 1915, it does not transmit error messagepacket 1916 to the switch 1902, rather it forwards an error message 1917to the upper layer. As can be seen from FIGS. 19A and 19B, the switchesand data store devices merely generate and forward error message packet,and do not perform error handling to correct the error.

[0127]FIG. 19C is a logic diagram illustrating the processing of acommunications node detecting an error in one embodiment. In block 1921,a component of a transport layer receives an error signal from the linklayer such as from a line indicating a CRC error. In block 1922, thecomponent retrieves packet identifier information associated with theerror. The packet identifier information includes the address of theinitiating host. In block 1923, the component generates an error packetto transmit to the initiating host. In block 1924, the componenttransmits the error packet to the initiating host.

[0128] 1.7 Transition Control & Selective Block Inversion

[0129] In one embodiment, the link layer of a communications nodeencodes codes into symbols in order to optimize transition control andto ensure DC-balance. Transition control may be optimized to eithermaximize or minimize the number of transitions in symbols that are to betransmitted. A transition is defined as a bit value of 1 followed by abit value of 0 or a bit value of 0 followed by a bit value of 1. Whenthe number of transitions is maximized, clock information may be moreaccurately derived from the transmitted symbols. When the number oftransitions is minimized, overall electromagnetic interference may beminimized. One skilled in the art will appreciate that the link layermay be designed to minimize or maximize transitions depending on theoverall design objectives. In the following, the described encodingtechnique attempts to maximize the number of transitions in symbols tobe transmitted. In an alternate embodiment, the transitions can bemaximized at times and minimized at other times to balance theobjectives of deriving a clock signal and reducing electromagneticinterference.

[0130] The encoding technique in one embodiment encodes an 8-bit codebased on the number of transitions in that code. The maximum possiblenumber of transitions in an 8-bit code is seven and occurs when the bitvalues alternate between 0 and 1 (e.g., “10101010”). The minimumpossible number of transitions in an 8-bit code is zero and occurs whenall the bit values are the same (e.g., “11111111”). To maximize thenumber of transitions, the encoding technique inverts the bit value ofalternate bits in each code with less than four transitions. Thisalternate bit inversion, also referred to as transition inversion,results in a code that previously had “n” transitions to now have [n-7]transitions. For example, the result of alternate bit inversion of acode with zero transitions (e.g., “11111111”) is a code with seventransitions (e.g., “01010101”), and the result of alternate bitinversion of the code with three transitions (e.g., “00010111”) is acode with four transitions (e.g., “10111101”). The encoding techniqueadds a ninth bit (i.e., a transition inversion indicating bit) to the8-bit code to indicate whether the alternate bits of the code wereinverted, which results in a 9-bit symbol. For example, a bit value of 0in the ninth bit indicates that the code did not have its alternate bitsinverted, and a bit value of 1 in the ninth bit indicates that the codedid have its alternate bits inverted. The symbol “010111101” has a bitvalue of 0 in its ninth bit, which indicates that alternate bits werenot inverted. In contrast, the symbol “110111101” has a bit value of 1in its ninth bit, which indicates that alternate bits were inverted. Oneskilled in the art will appreciate that the ninth bit can be stored atany bit position within the symbol. In this description, the ninth bitis represented as the most significant bit of the symbols.

[0131] When a symbol is received at a communications node, the ninth bitis used to determine whether the symbol needs to have its alternate bitsinverted to decode the symbol. Since the symbol “010111101” has a bitvalue of 0 in its ninth bit, its alternate bits were not inverted andthe resulting decoded code is “10111101.” Conversely, since the symbol“110111101” has a bit value of 1 in its ninth bit, its alternate bitswere inverted and the resulting decoded code is “00010111.” All thesymbols generated by alternate bit inversion encoding are referred to as“in-band symbols” because they are 9-bit symbols with at least fourtransitions. All 9-bit symbols with three or less transitions arereferred to as “out-of-band symbols.” This alternate bit inversionencoding technique generates only in-band symbols.

[0132]FIG. 20 illustrates an alternate bit inversion encoding techniquein one embodiment. The left column represents the 8-bit codes, and theright column represents the 9-bit symbols. The alternate bit inversionencoding technique is referred to as an 8 b/9 b encoding, which is amore efficient encoding than standard 8 b/10 b encodings. Transitioncontrol encoding is described in U.S. Pat. No. 5,999,571, entitled“Transition-Controlled Digital Encoding and Signal Transmission System,”which is hereby incorporated by reference.

[0133] The encoding technique may also help ensure DC-balance by usingselective block inversion encoding on symbols that have been transitionoptimized. Selective block inversion refers to an encoding techniquethat inverts certain bit values in a block of symbols to help ensureDC-balance is maintained. In one embodiment, selective block inversionmay perform polarity inversion (i.e., inverting each bit of a symbol) onall but one symbol of a block to help ensure DC-balance and transitioninversion on that one symbol in the block to indicate that the othersymbols of the block have been polarity inverted. Although the encodingtechnique performs selective block inversion on blocks of four symbols,one skilled in the art will appreciate that a block can contain anynumber of symbols. Conventional encoding techniques, in contrast,typically process only one symbol at a time when attempting to ensureDC-balance. Because the selective block inversion encoding techniqueevaluates DC-balance over multiple symbols, DC-balance is more likelyassured and there is essentially no bandwidth used to indicate whether asymbol has been inverted.

[0134] The selective block inversion encoding technique tracks therunning disparity on symbol boundaries of the symbols that have beentransmitted so far. The running disparity represents the difference inthe overall number of 1-bit values and 0-bit values of the symbols thathave been transmitted on the communications link. For example, if 4521-bit values and 448 0-bit values have been transmitted (e.g., 100symbols), then the running disparity is +4 (e.g., in the 1-direction orpositive). If 448 1-bit values and 452 0-bit values have beentransmitted, then the running disparity is −4 (e.g., in the 0-directionor negative). If the number of 1-bit values and 0-bit values are equal,then the running disparity is 0 (e.g., neutral). To determine whether toperform selective block inversion on a block of symbols, the encodingtechnique compares the running disparity of the symbols transmitted sofar to the block disparity of that block. Block disparity is thedifference between number of 1-bit values and 0-bit values of thesymbols of a block. If the running disparity and the block disparity areboth positive or both negative, then the encoding technique performsblock inversion on that block.

[0135] The encoding technique inverts a block by inverting alternatebits of the first symbol in the block (i.e., transition inversion) andinverting all the bits of the other symbols in the block (i.e., polarityinversion). The inverting of the alternate bits of the first symbolresults in an out-of-band symbol that is used to indicate whether thefollowing block has been block inverted. When a block of symbols isreceived, the receiving communications node can determine whether theblock has been block inverted by checking the first symbol of the block.If the first symbol of the block is an in-band symbol, then the blockhas not been block inverted. If the first symbol of the block is anout-of-band symbol, then the block has been block inverted. If the blockhas been block inverted, then the decoding technique undoes thealternate bit inversion of the first symbol by inverting alternate bitsof the first symbol and undoes the bit inversion of the other symbols byinverting all of the bits of the other symbols. One skilled in the artwill appreciate that the symbol used to indicate whether the block isblock inverted need not be the first symbol of the block, but rather canbe any symbol of the block so long as the receiving communications nodecan identify which symbol indicates selective block inversion.

[0136]FIG. 21A is a diagram illustrating the selective block inversionencoding technique in one embodiment. In this example, a block isdefined as having four symbols. One skilled in the art will appreciatethat a block can have any number of two or more symbols. In thisexample, the block disparity is +6, because there are 21 1-bit valuesand 15 0-bit values in the block. If the running disparity is positive,then the encoding technique performs block inversion. The first symbolof the block has its alternate bits inverted, and the remaining symbolsof the block have all their bits inverted. The resulting symbols have ablock disparity of −4. Thus, the DC-balance would be improved bytransmitting the inverted block, rather than the noninverted block. Theencoding technique may alternatively only perform block inversion whenthe running disparity is outside a certain range. For example, if therunning disparity is between −100 and +100, then no selective blockinversion is performed. If outside the range, then selective blockinversion is performed. One skilled in the art will appreciate that thisrange can be selected based on various characteristics of thecommunications link such as transmission rate or maximum transmissiondistance.

[0137]FIG. 21B is a block diagram illustrating selective block inversionwith a programmable block size in one embodiment. In this example, theblock size is represented by the variable “k.” The selective blockinversion encoding technique processes k sequential symbols at a time asindicated by blocks 2101 and 2102. The transition from block 2101 to2102 illustrates selective block inversion for transmission, and thetransition from block 2102 to 2101 illustrates the undoing of selectiveblock inversion upon reception. To perform block inversion, the blockheader symbol Ci is transition inverted (TI) resulting in symbol Bi andthe symbol Ci+1 to symbol Ci+k−1 are polarity inverted (SI) resulting insymbol {overscore (C_(i+1))} to symbol {overscore (C_(i+k−1))}. To undoblock inversion, the block header symbol Bi is transition invertedresulting in symbol Ci and the symbols {overscore (C_(i+1))} to symbol{overscore (C_(i+k−1))} are polarity inverted resulting in symbol Ci+1and symbol Ci+k+1.

[0138]FIG. 21C is a block diagram illustrating transition inversion andpolarity inversion in one embodiment. In this example, 8-bit codes areinverted. The code 2110 may be transition inverted in two ways togenerate codes 2111 and 2112. Code 2111 results from inverting alternatebits starting with the first bit, and code 2112 results from invertingalternate bits starting with the second bit. The code 2110 is polarityinverted by inverting each bit to generate code 2113. The same processis applied to codes 2111, 2112, and 2113 to generate code 2110.

[0139] As discussed above, a primitive is a sequence of two out-of-bandsymbols. Thus, a receiving communications node can distinguish between aprimitive and the first symbol of an inverted block because a primitiveis a sequence of two out-of-band symbols and one out-of-band symbol byitself represents the first symbol of an inverted block.

[0140]FIG. 22 is a diagram illustrating the format of a primitive in oneembodiment. A primitive is defined as a sequence of two out-of-bandsymbols. The out-of-band symbols are preferably selected to have minimalimpact on DC-balance. In particular, the disparity of a primitive isselected to be as close to 0 as possible. To achieve this disparity, onesymbol of the primitive may have a negative disparity and the othersymbol of the primitive may have any positive disparity. (A symbol withan odd number of bits cannot have a 0 disparity.) When the disparity ofa primitive is 0, then the primitive can be transmitted at any timewithout affecting the running disparity and thus with minimal impact onthe DC-balance. In one embodiment, the first symbol of a primitive hasthe negative disparity (i.e., negative polarity) and the second symbolhas the positive disparity (i.e., positive polarity), or vice versa. Thereceiving communications node can detect a transmission error when asequence of two out-of-band symbols has an improper ordering for theirdisparity or both symbols have either a positive or a negativedisparity. When a primitive has minimal or no effect on DC-balance, thenthe complexity of the transmitting communications node may be reducedbecause the logic to calculate the running disparity need not includeprimitives in the calculation.

[0141] FIGS. 23-28 are logic diagrams illustrating encoding and decodingtechniques in one embodiment. FIG. 23 is a logic diagram illustratingtransition control and selective block inversion encoding in oneembodiment. A component of the link layer receives codes to betransmitted one code at a time from the transport layer. In block 2301,the component performs transition control optimization to generate asymbol from the code. In block 2302, the component buffers the symbolsuntil a complete block of symbols has been buffered. In decision block2303, if a complete block of symbols has been buffered, then thecomponent continues at a block 2304, else the component waits to receivethe next code. In decision block 2304, if the running disparity is equalto the block disparity, then the component continues at block 2305, elsethe component continues at block 2306. In block 2305, the componentperforms block inversion on the symbols of the block. In block 2306, thecomponent provides the symbols of the buffer to the physical layer andthen waits to receive the next code from the transport layer.

[0142]FIG. 24 is a logic diagram illustrating transition controloptimization in one embodiment. In block 2401, the component receives acode for transition optimization. In block 2402, the component countsthe number of transitions in the received code. In block 2403, if thenumber of transitions is less than four (assuming an 8-bit code), thenthe component continues at block 2405, else the component continues atblock 2404. In block 2404, the component sets the ninth bit to a 0-bitvalue. In block 2405, the component performs alternate bit inversion onthe received code by inverting every other bit. In block 2406, thecomponent sets the ninth bit to a 1-bit value. In block 2407, thecomponent generates the symbol as the ninth bit value plus the code.

[0143]FIG. 25 is a logic diagram illustrating block inversion in oneembodiment. In block 2501, the component selects the first symbol of theblock. In block 2502, the component performs alternate bit inversion onthe selected symbol by inverting alternate bits. In block 2503, thecomponent selects the next symbol of the block. In decision block 2504,if all the symbols have already been selected, then the componentcompletes, else the component continues at block 2505. In block 2505,the component inverts the selected symbol by inverting the value of eachbit in the selected symbol. The component then selects the next symbolof the block.

[0144]FIG. 26 is a logic diagram illustrating transition control andselective block inversion decoding in one embodiment. The component ofthe link layer is provided with one symbol at a time by the physicallayer. In block 2601, the component stores the symbol in a buffer. Indecision block 2602, if the buffer contains a primitive, then thecomponent continues at block 2603 to process the primitive, else thecomponent continues at block 2604. In block 2604, if the buffer containsa complete block, then the component continues at block 2605, else thecomponent waits for the next symbol. In decision block 2605, if thefirst symbol in the buffer is an out-of-band symbol, then thetransmitting communications node inverted the block and the componentcontinues at block 2606, else the component continues at block 2607. Inblock 2606, the component undoes the block inversion of the block. Inblock 2607, the component selects the next symbol of the block. Indecision block 2608, if all the symbols have already been selected, thenthe component waits for the next symbol, else the component continues atblock 2609. In block 2609, the component undoes the optimization of thetransitions of the selected symbol to generate the corresponding code.In block 2610, the component provides the code to the transport layerand then processes the next symbol of the block.

[0145]FIG. 27 is a logic diagram illustrating the undoing of blockinversion in one embodiment. In block 2701, the component selects thefirst symbol of the block. In block 2702, the component invertsalternate bits of the selected symbol. In block 2703, the componentselects the next symbol of the block. In decision block 2704, if all thesymbols have already been selected, then the component completes, elsethe component continues at block 2705. In block 2705, the componentinverts all the bits of the selected symbol and then processes the nextsymbol of the block.

[0146]FIG. 28 is a logic diagram illustrating the undoing of transitioncontrol optimization in one embodiment. In decision block 2801, if theninth bit of the symbol is one, then the component inverts alternatebits of the symbol in block 2802. The component then outputs theresulting code.

[0147]FIG. 29 is a logic diagram illustrating the processing of aprimitive in one embodiment. Although not shown, the component mayensure that the disparity of the symbols of the primitive are in thecorrect order. In decision blocks 2901-2903, the component decodes theprimitive. In blocks 2904-2906, the component raises a signal indicatingthe primitive that has been received. In one embodiment, the signals maybe provided to the transport layer to control the packetizing of thereceived codes.

[0148] 2. Memory Link Architecture

[0149] In one embodiment, various transport layer, link layer, andphysical layer techniques can be used to provide a serial communicationsarchitecture that is well suited to support memory-based data storedevices, especially multiport memory devices with random access memory.This architecture that is adapted to memory devices is referred to as“Memory Link.” A multiport memory device provides multiple serialcommunications ports that can be connected to different hosts (e.g., anymemory accessing device, such as a memory controller). Each host canindependently access the memory of the memory device to read and writedata. The memory may be organized into memory banks that can besimultaneously accessed by the same or different hosts. Manyapplications can advantageously use a multiport memory device. Forexample, an asynchronous transfer mode (“ATM”) switch can use amultiport memory device to replace a crossbar switch. The ATM switch mayhave multiple input ports that store received data in the multiportmemory device and multiple output ports that retrieve data from themultiport memory device. A multiport memory device may include a switchfor dynamically interconnecting the memory to the ports so that eachport can selectively access each memory bank. Alternatively, the portsmay access memory using time-division multiplexing techniques.

[0150] Because of the way memory is typically accessed, the physicallayer, link layer, and transport layer might be simplified as comparedto the layers needed in a more general data store device, such as a diskdrive. Each port of a multiport memory device is typically directlyconnected to a host. In such a case, packets do not need to specify asource and a destination and the headers are thus simplified. Inaddition, packet types can be encoded in synchronization primitives tofurther simplify the packet header. The transport layer of a port in amultiport memory device can be simplified or eliminated because (1)routing to another device is not needed, (2) minimal or no parsing ofthe header may be needed, and (3) preemption and priority ordering ofpackets may not be needed. In addition, the link layer may be simplifiedby the use of physical layer DC-balancing, rather than the use of theselective block inversion to achieve DC-balance. Physical layerDC-balancing can be used with a multiport memory device because thedistance between the host and the multiport memory device is typicallyrelatively short.

[0151] The physical layer may also be simplified because a singlemultiphase clock generating device, such as an analog phase lock loop(“PLL”), or a delay lock loop, can be used to provide clocking for thetransmitter and receiver for each port of the multiport memory device.The use of a single multiphase clock generating device can greatlyreduce the area of a multiport memory chip used by the physical layer. Asingle-chip, multiport memory device has traditionally been infeasiblebecause of the need to have multiple analog phase lock loops. Thephysical layer may also use plesiosynchronous null insertion techniquesas described below to compensate for variations in the frequency betweenclocks of the hosts and the clock of the multiport memory device. Themultiport memory device may transmit control information usingout-of-band symbols as described above. Also, commands for accessing thememory device may be encoded in synchronization symbols.

[0152]FIG. 30 is a block diagram illustrating the overall architectureof a multiport memory device in one embodiment. The multiport memorydevice 3000 includes multiple ports 3010-3019. Each port includes aphysical layer, a link layer, a transport layer (optional), and anaccess layer. The access layers are connected to switch 3050. Themultiport memory device includes memory with memory banks 3320-3027.Each memory bank has associated bank cache 3030-3037. The bank cachesare connected to the switch. A switch arbiter (not shown) is responsiblefor connecting the ports to the bank caches as needed to process thememory access requests. Each port can be independently andsimultaneously connected to memory banks. Alternatively, access of theports to the memory banks can be time-division multiplexed. Withtime-division multiplexing, each port has its allotted time period forcommunicating with the memory bank. A clock signal with a differentphase for each port can be used to coordinate the access. The physicallayer, link layer, and transport layer may be implemented to providevarious combinations of the functions described above (e.g., transitioncontrol optimization and synchronization primitives encoded with packettypes). The access layer receives commands and data from the transportlayer, directs the arbiter to connect the port to the appropriate memorybank, and transmits commands and data to the memory bank. The accesslayer also receives data from the memory banks and provides the data tothe transport layer for transmission to the host.

[0153]FIG. 31 is a block diagram illustrating the use of a single phaselock loop for the physical layer of a multiport memory device in oneembodiment. The physical layer 3100 of the multiport memory deviceincludes multiple ports 3110-3119. Each port includes a serialcommunications line connected to a transmitter and another serialcommunications line connected to a receiver that comprise the serialcommunications link. Each port also includes a serializer anddeserializer. The phase lock loop 3120 receives a clock signal fromclock 3121. The phase lock loop outputs a clock signal 3122 and phaseshifted clock signals 3123. The clock signal 3122 provides clockinginformation for the serializer. The phase shifted clock signals 3123 areprovided to the deserializers to control the sampling of the input data.

[0154]FIG. 32 is a block diagram illustrating contents of a bank cachein one embodiment. Each bank cache includes an input queue 3201 and anoutput queue 3202. Each entry of the input queue includes a port field,a read/write field, an address field, and a data field. The port fieldidentifies the port (e.g., 3) that provided the command. The read/writefield identifies whether the access is a read or a write (i.e., R or W).The address field identifies the memory address to be accessed (e.g.,1000). The address field may be latched from the address bus. The datafield includes the data to be written to memory. Each entry of theoutput queue includes a valid field, a port field, and a data field. Thevalid field indicates whether the entry contains valid data (e.g., 0 or1). If a memory bank has multiple sections as described below, then thevalid field may have a bit indicating whether the data of each sectionis valid. The port field identifies the port (e.g., 3) to which the datais to be transmitted. The data field contains the data retrieved frommemory. Each memory bank successively retrieves entries from the inputqueue and then performs the requested access. When the access is to readdata from memory, the memory bank also adds entries to the output queue.The memory bank adds an entry to the output queue on every possible readcycle. The valid field is used to indicate whether the entry containsvalid data.

[0155]FIG. 33 is a logic diagram illustrating processing of the accesslayer when it receives data from a host in one embodiment. In block3301, a component of the access layer receives the next command from thetransport layer. In decision block 3302-3304, the access layer parsesthe command. In block 3305-3307, the component performs the processingassociated with that command.

[0156]FIG. 34 is a logic diagram illustrating the processing of a writecommand by the access layer in one embodiment. In block 3401, thecomponent retrieves the target address for the write command. In block3402, the component requests the arbiter to configure the switch toconnect the port with the appropriate memory bank. In block 3403, thecomponent retrieves the next byte (i.e., code) from the transport layer.In decision block 3404, if all the bytes have already been retrieved,then the write is complete, else the component continues at block 3405.In block 3405, the component stores an entry into the input queue of thememory bank. The component may need to buffer multiple bytes dependingon the width of the word. In block 3406, the component increments theaddress and then reads the next byte. One skilled in the art willappreciate that when time-division multiplexing is used, the componentsupplies the data to the input queue during its allotted time period.

[0157]FIG. 35 is a logic diagram illustrating processing of the accesslayer when it is to provide data to the transport layer in oneembodiment. In block 3501, the component receives the next byte from theswitch. In decision block 3502, if all the bytes have already beenreceived, then the component continues at block 3504, else the componentcontinues at block 3503. In block 3503, the component provides the byteto the transport layer and then proceeds to receive the next byte. Inblock 3504, the access layer signals the end of the read to thetransport layer.

[0158] 2.1 Multisection Memory Banks

[0159]FIG. 36 is a block diagram of a multisection memory bank in oneembodiment. A multisection memory bank can significantly reduce powerconsumption of a memory device. A multisection memory bank subdivideseach row (or word) of data in a memory bank into multiple sections. Theenabling of sections can be configured on a port-by-port basis. When asection is disabled, then the amount of power consumed by the memorybank during an access is much less than if the section was enabled. Inthis way, each port can specify only the needed sections which canresult in a significant reduction in the power consumption because onlythe needed sections are enabled during access. The data for all the rowsfor each subdivision of a memory bank is stored in a section (or apage). For example, a memory bank may have a word width of 64 bits. Sucha multisection memory bank may be divided into two sections of 32 bitseach that can be accessed independently and simultaneously based onconfigurable parameters. If a host only needs a 32-bit word, then onlyone section needs to be enabled whenever that host accesses the memorybank.

[0160] As shown in FIG. 36, the multisection memory bank includessection 3600 and section 3610 each of which is 32 bits wide. Eachwordline of each section includes an AND gate 3601, 3611 that isconnected to a corresponding line of the row decoder 3630 and to asection enable line from configuration register 3650. The row decoderdecodes the row portion of the address and enables the appropriatewordline when the corresponding section is enabled. For example, if therow portion of the address is eight bits, then the row decoder enablesone of 256 wordlines. The AND gate enables the wordline for its sectiononly when that section is enabled for the port. If a port only accessessection 3600, then whenever an access for the port is processed, section3600 is enabled and section 3610 is disabled. Each wordline of eachsection may also include a sequence of latches 3602, 3612 for bufferingsignals. The number of latches needed depends on row access latency,column access latency, and desired level of simultaneous access. Eachsection also includes a column decoder and column cache 3603, 3613. Thecolumn decoder latches data into the column cache based on the columnaddress provided by the sequence of latches 3104, 3114 for bufferingcolumn address. Because the wordlines are only enabled on asection-by-section basis and because retrieved data is only latched on asection-by-section basis, the power requirements of a multisectionmemory bank is lower than a memory bank that does not have multiplesections. When data is retrieved from a multisection memory bank, thevalid field of the output queue indicates the divisions of the word thatare valid. For example, if data is retrieved from only section 0, thenthe valid field may contain “10.” If data is retrieved from bothsections 0 and 1, then the valid field may contain “11.” One skilled inthe art will appreciate that a memory bank may be subdivided into anumber of sections. Also, a multisection memory bank may simultaneouslyperform multiple access commands so long as the commands do not accessthe same wordline of the same section. Thus, the processing of a commandto access wordline 10 of section 0 may overlap with the processing of acommand to access wordline 11 of section 0. Also, the processing of acommand to access wordline 10 of section 0 may be overlapped with theprocessing of a command to access wordline 10 of section 1. Thus,increasing the number of sections of a memory bank can increase theamount of parallel processing that can be performed.

[0161] 2.2 Physical layer DC-balancing

[0162] In one embodiment, DC-balancing of the communications link ishandled at the physical layer in memory link. The performing ofDC-balancing at the physical layer can help reduce the cost andcomplexity of the link layer. To help ensure DC-balance, the physicallayer uses a variable strength line driver that can drive a standardstrength or increased strength 0-bit value or 1-bit value onto thecommunications link. When the running disparity is zero or positive(i.e., in the 1-direction) and a 1-bit value is to be transmitted, thena standard strength 1-bit value is driven. Similarly, when the runningdisparity is a zero or negative (i.e., in the 0-direction) and a 0-bitvalue is to be transmitted, then a standard strength 0-bit value isdriven. In contrast, when the running disparity is negative and a 1-bitvalue is to be transmitted, then an increased strength 1-bit value(i.e., a 1-bit voltage value) is driven. This increased strength 1-bitvalue tends to offset the effects of a negative running disparity.Similarly, when the running disparity is positive and a 0-bit value isto be transmitted, then an increased strength 0-bit value (i.e., a 0-bitvoltage value) is driven. This increased strength 0-bit value tends tooffset the effects of a positive running disparity. The use of avariable-strength line driver is especially useful in controllingDC-balance over short transmission distances. In such a case, physicallayer DC-balancing can be used in lieu of link layer DC-balancing.

[0163]FIG. 37A is a block diagram illustrating a variable strength linedriver in one embodiment. A variable strength line driver 3700 includesa fixed-strength component (or portion) 3701 and a variable-strengthcomponent 3702. The variable-strength line driver has 4 inputs: data in3703, negative running disparity signal (RD−) 3705, positive runningdisparity signal (RD+) 3706, and enable signal 3707. Thevariable-strength line driver has one output: data out 3704. Thefixed-strength component outputs a standard strength 0-bit value or1-bit value depending on data in. The variable-strength component pullsup, pulls down, or has no effect on data out, depending on the runningdisparity and data in. If the running disparity is positive and data inis a 0-bit value, then the variable-strength component pulls down dataout. If the running disparity is negative and data in is a 1-bit value,then the variable-strength component pulls up data out. More generally,the running disparity signals may indicate when the running disparity issufficiently negative or sufficiently positive to activate thevariable-strength component.

[0164]FIG. 37B is a block diagram illustrating the circuit of avariable-strength line driver in one embodiment. The fixed-strengthcomponent 3701 and the variable-strength component 3702 employ the samelogic to activate their pull-up and pull-down transistors. One skilledin the art will appreciate that the transistors may be selected toachieve the overall desired physical layer characteristics of thecommunications link. NAND gate 3710 is used to control pull-uptransistor 3711, and NOR gate 3712 is used to control pull-downtransistor 3713. When the data in is a 1-bit value and the driver isenabled, NAND gate 3710 outputs a 0, which turns on pull-up transistor3711. When the data in is a 0-bit value and the driver is enabled, NORgate 3712 outputs a 1, which turns on pull-down transistor 3713. Thevariable-strength component has similar control logic, except thatpull-up transistor 3721 is activated only when the running disparity isnegative and except that the pull-down transistor 3723 is activated onlywhen the running disparity is positive.

[0165] 3. Plesiosynchronous Clocking with NULL Insertion/Removal

[0166] In one embodiment, the serial communications architecturespecifies that NULL symbols or NULL primitives are inserted into orremoved from a stream of symbols to account for variations in clockfrequency between the transmitting and receiving communications nodeswhen using plesiosynchronous clocking. If a transmitting communicationsnode transmits at a slightly higher frequency than the receivingcommunications node receives, then the receiving communications nodetraditionally needs to buffer the received symbols to handle such anoverrun. For example, if both communications nodes use clocks with afrequency of 100 Hz with a maximum variation of 1%, then it is possiblethat the transmitting communications node transmits at 101 Hz and thereceiving communications node receives at 99 Hz. In such a case, thereceiving communications node will receive an extra 2 bits approximatelyevery second, assuming 1 bit is transmitted every clock cycle. Receivingcommunications nodes typically have an elastic buffer for storing andaligning these extra bits on symbol boundaries. Similarly, an underrunoccurs when a transmitting communications node transmits data at aslightly lower rate than the receiving communications node receivers.The handling of an underrun can also be costly and complex. Theseunderrun and overrun conditions were typically handled at the linklayer. These complexities and costs may, however, be avoided by thephysical layer of a receiving communications node periodically insertingor removing NULL symbols or NULL primitives from the received stream ofsymbols at a rate sufficient to compensate for variations in clockfrequencies.

[0167]FIG. 38A is a block diagram of a plesiosynchronous communicationssystem in one embodiment. Communications node 3801 and communicationsnode 3811 are connected via a bi-directional serial link. The transportlayer and the link layer 3802, 3812 may be implemented in a singledigital ASIC chip with other interface logic such as PCI bus interfacelogic. The physical layer 3803 and 3813 may be built in a separatedcustom analog chip.

[0168] The link layer provides symbols (i.e., N-bit encoded data) to betransmitted to the physical layer via TX_DAT lines. The link layer isprovided symbols received by the physical layer via the RX_DAT lines. Inone embodiment, the TX_DAT and RX_DAT lines are both synchronized to thelocal clock, and the setup/hold time relation between TX_DAT and RX_DATare the same to setup/hold time relationship to the local clock. Boththe transmitter 3804, 3814 and receiver 3805, 3815 of the physical layershare the same phase lock loop (“PLL”) 3806, 3816. This overallarchitecture is described in U.S. patent application Ser. No. 6,229,859,entitled “System and Method for High-Speed, Synchronized DataCommunication.”

[0169]FIG. 38B is a block diagram illustrating a switch supportingmultiple data store devices in one embodiment. The switch 3820 has 5ports 3821-3825. Ports 3821-3824 are shown connected to disk drives3831-3834. The switch has one phase lock loop 3826 that is shared by allthe ports. Since all the ports of the switch are synchronized to thesame clock, the data can be freely (without buffering) transferredbetween ports. (See discussion below on asynchronous clock boundaries.)

[0170]FIG. 39A is a block diagram illustrating a prior art synchronousclocking system. In this synchronous clocking system, the RX_DAT linesprovided to the link layer 3920 are synchronized not to the local clock,but to the RX_CLK line provided by the physical layer. The RX_CLK lineprovides a clock that has been recovered from the incoming bit stream,which is synchronized to the transmitter's clock. The receiver mayrecover the clock from a clock signal on a separate line or from thestream of data bits transmitted by the transmitter. In the link layer,there are two clock domains, the TX clock domain 3921 that is the sameas the local clock domain, and the RX clock domain 3922 that is the sameas the transmitter's clock domain. Consequently, an asynchronous clockboundary 3923 is formed between the TX clock domain and the RX clockdomain. When information is transferred from the TX clock domain to theRX clock domain, the FIFO 3924 is used for the data and flip-flopsynchronizer 3925 is used for the control information. When informationis transferred from the RX clock domain to the TX clock domain, the FIFO3926 is used for the data and the flip-flop synchronizer 3927 is usedfor the control information.

[0171]FIG. 39B is a block diagram illustrating resolution of anasynchronous clock boundary in the physical layer in one embodiment. Theclock and data recover (“CDR”) processing of the physical layer 3950provides to the link layer RX_DAT lines that are synchronized to thelocal clock. Thus, the asynchronous boundary at the link layer alongwith the needed logic to resolve the boundary is avoided in the linklayer. The CDR processing also provides a 1-bit control signal(DET_NULL) that indicates a NULL symbol is provided in RX_DAT (i.e., toinform the link layer that a primitive has been received) and a 1-bitcontrol signal (DET_ERR) that indicates an error in the physical layerdata recovery.

[0172]FIG. 40 is a block diagram illustrating NULL insertion anddeletion at the physical layer in one embodiment. Sampler 4010 in oneembodiment corresponds to the over sampler 4011, digital phase lock loop4012, and phase selector 4013 described in U.S. Pat. No. 6,229,859,entitled “System and Method for High-Speed, Synchronized DataCommunication.” The sampler outputs N+1 bits (where N is the number ofbits in a symbol) of data lines D[0:N] and two valid bits V[N−1] andV[N]. The sampler identifies a stream of data bits that are received andprovides those data bits in data lines D[0:N]. The valid bits indicatewhether D[N−1] and D[N] contain valid data (e.g., V[N−1]=1 indicatesthat D[N−1] is valid). The data bits provided by the sampler are not,however, frame (i.e., symbol) aligned; that is, D[0] is not necessarily(and more often than not would not be) the first bit in a symbol. Thevalid bits indicate whether an overrun or underrun condition hasoccurred. An overrun occurs as a result of the transmitter's clockfrequency being faster than the receiver's, and an underrun occurs as aresult of the transmitter's clock frequency being slower than thereceiver's. Nominally, N data bits are provided by the sampler everysample interval. In such a case D[N−1] is valid, but D[N] is invalid.If, however, there is an overrun, then the sampler provides N+1 databits; that is, D[N−1] and D[N] are both valid. If there is an underrun,then the sampler provides N−1 data bits; that is, D[N−1] and D[] Nareboth invalid. The data bits and the valid bits are supplied to a framealigner 4020. The frame aligner not only aligns the frames, but alsoinserts and removes NULL symbols to adjust for variations in thetransmitting and receiving clock frequencies.

[0173]FIG. 41 is a block diagram illustrating the frame aligner in oneembodiment. The frame aligner 4100 includes bit buffer 4101, sync andnull detector 4103, symbol pointer tracker 4104, and multiplexer 4102.The bit buffer stores the last 27 bits (e.g., 3×N) received as providedby D[0:9], assuming that a symbol includes 9 bits. The bit buffer isorganized into three buffers or regions (i.e., A, B, and C) of 9 bitseach. The sync and null detector scans the bit buffer to detect whetherit contains a NULL primitive and, in particular, a synchronizationprimitive. If a synchronization primitive is detected, it signals thesymbol pointer tracker to align itself on the synchronization primitive.The sync and null detector also signals the symbol pointer trackerwhether a NULL symbol should be inserted or removed from the symbolstream. The symbol tracker pointer tracks the symbol alignment overtime.

[0174] The sync and null detector provides a 1-bit load data signal(“LD”), a 1-bit last half signal (“iHF”), and a 9-bit synchronizationsymbol pointer (“iPTR”) to the symbol pointer tracker. The LD signalindicates that a synchronization symbol has been detected. The iHFsignal indicates whether the synchronization symbol starts in the firstor last half of one of the 9-bit buffers of the bit buffer. The iPTRsignal indicates a bit position of the start of the synchronizationprimitive in the bit buffer.

[0175] The symbol pointer tracker provides 1-bit null selection signal(“uHF”) and 9-bit start-of-symbol pointer (“uPTR”) to the multiplexer.The uHF signal indicates whether to insert a NULL symbol into the streamof symbols, and the uPTR signal indicates the start bit position of asymbol. The symbol pointer tracker also provides 1-bit error signal(“DET.ERR”) and 1-bit null signal (“DET.NULL”) to the link layer. Themultiplexer provides the 9-bit RX_DAT to the link layer.

[0176]FIGS. 42A, 42B, and 42C illustrate the loading of the bit bufferin one embodiment. The bit buffer contains three 9-bit buffers A[8:0],B[8:0], and C[8:0]. When the valid bits V[8:9] indicate that only D[8]is valid (i.e., V[8:9]=“10” a normal condition), then all the data inthe bit buffer is rotated 9 bits to the left. In other words, the datafrom B[8:0] is transferred to A[8:0], the data from C[8:0] istransferred to B[8:0], and the data from D[0:8] is transferred to C[8:0]as indicated by FIG. 42A. When the valid bits V[8:9] indicate thatneither D[8] or D[9] is valid (i.e., V[8:9]=“00” an underrun condition),then the data in the bit buffer is rotated 8 bits to the left. In otherwords, A[0] is transferred to A[8], B[8:1] is transferred to A[7:0],B[0] is transferred to B[8], C[8:1] is transferred to B[7:0], C[0] toC[8], and D[0:7] is transferred to C[7:0] as illustrated by FIG. 42B.When the valid data V[8:9] indicates that both D[8] and D[9] are valid(i.e., V[8:9]=“11” an overrun condition), then the data in the bitbuffer is rotated 10 bits to the left. In other words, B[7:0] istransferred to A[8:1], C[8] is transferred to A[0], C[7:0] istransferred B[8:1], D[0] is transferred to B[0], and D[1:9] istransferred to C[8:0] illustrated by FIG. 42C.

[0177]FIG. 43 is a diagram illustrating the detection of asynchronization primitive by the sync and null detector in oneembodiment. Assuming that the synchronization detection mode is fallingedge, the detector determines whether there are only three falling edgesin the bit buffer and whether they are 9 bits apart. If so, the detectorsets the LD signal to 1, else the detector sets the LD signal to 0. Ifthe falling edges are in the first half of the bit buffer (e.g.,A[8:5]), then the detector sets the iHF signal to 0, else the detectorsets the iHF signal to 1. The detector also sets the iPTR signals toindicate the position of the falling edge that starts thesynchronization symbols, which indicate the start bit position of thesynchronization symbol. In this example, the bit buffer 4301 containsthe synchronization primitive “100000001-100111111” starting at A[6] anda portion of that synchronization primitive starts at C[6]. If A[8:7]contained a falling edge from a non-synchronization symbol, then whenthe next symbol of the second synchronization primitive is received fromthe sampler, the bit buffer would be guaranteed to contain only threefalling edges. The bit buffer 4302 contains the same synchronizationprimitives.

[0178]FIG. 44 is a diagram illustrating the portion of the bit bufferthat contains the next symbol to be extracted in one embodiment. If thestart of a symbol is in the first half of a region of the bit buffer(i.e., A[8:5]), then the symbol is extracted starting in the B region ofthe bit buffer and proceeding to the C region. If the start of a symbolis in the last half of a region of the bit buffer (i.e., A[4:0]), thenthe symbol is extracted starting in the A region of the bit buffer andproceeding to the B region.

[0179]FIG. 45 is a state diagram of the symbol pointer tracker in oneembodiment. The symbol pointer tracker tracks the start position of asymbol (e.g., frame) based on the start position of a synchronizationsymbol and the valid bits (e.g., V[8:9]) provided by the sampler. Thesymbol pointer tracker enters an initial state (“INIT”) whenever the LDsignal of the sync and null detector indicates that a synchronizationsymbol has been detected. The tracker stays in that initial state solong as the frame aligner is provided with 9 valid bits (i.e., N validbits) at a time (i.e., V[8:9]=“10” a normal condition). If the framealigner is provided with 10 valid bits (i.e., V[8:9]=“11” an overruncondition), then the tracker enters a first overrun state (“OVR1”),which means that the transmitter's clock frequency is faster than thereceivers clock frequency. The tracker stays in the first overrun stateso long as the frame aligner is provided with 9 valid bits at a time. Ifthe frame aligner is provided with 10 valid bits, then the trackerenters the second overrun state (“OVR2”) from the first overrun state.If the frame aligner is provided with 8 valid bits, then an error hasoccurred (e.g., 10 bits valid followed by 8 bits valid) and enters anerror state from the first overrun state. The tracker exits the secondoverrun state when the next bits are provided to the frame aligner. Ifthe tracker is provided with 9 valid bits, then the tracker returns tothe first overrun state, else the tracker enters an error state. Once anoverrun is detected, the tracker thus expects to receive only 9 or 10valid bits at a time and does not expect to receive two sets of 10 validbits in a row. The receiving of two sets of 10 valid bits in a row wouldindicate that the transmitter's clock frequency is too high to becompensated for by the receiver. The receiving of 8 bits in an overrunstate indicates that the transmitter's clock frequency now appears to beslower, which means some sort of error has occurred. The first underrunstate (“UND1”) and the second underrun state (“UND2”) are entered in amanner that is analogous to entry into the overrun states.

[0180] When the tracker detects an LD signal of 1, it sets thestart-of-symbol pointer to the start of the synchronization symbol asindicated by the iPTR lines and enters the initial state. It then setsthe extraction region for the start-of-symbol pointer to A if the iHFsignal is 1 and to B if the iHF signal is 0. The extraction regionindicates the region that contains the first bit of the next symbol tobe extracted. When the tracker detects an LD signal of 0, it moves thestart-of-symbol pointer either backward or forward, depending on whetherthe tracker transitions to an overrun state as a result of receiving 10bits or transitions to an underrun state as a result of receiving 8bits.

[0181]FIG. 46 is a diagram illustrating movement of the start-of-symbolpointer due to overrun or underrun in one embodiment. When asynchronization symbol is detected (i.e., LD=1), then thestart-of-symbol pointer is somewhere within the range of A[3]:B[4]. Ifthe transmitter's local clock frequency is slightly slower than thereceiver's local clock frequency (i.e., a potential underrun condition),the tracker moves the pointer forward to the C region. When the pointeris moved into the underrun extend region (e.g., B[3]:C[8]) and a NULLsymbol is detected, then the tracker moves the start-of-symbol pointerbackward 9 bit positions (i.e., 1 symbol position) and inserts a NULLsymbol at the start-of-symbol pointer. This insertion of the NULL symbolcompensates for the underrun by adding an extra symbol to the receivedstream. In one embodiment, a NULL symbol can be any symbol of anyprimitive, that is, control or synchronization. Similarly, if thetransmitter's local clock frequency is slightly faster than thereceiver's local clock frequency (i.e., a potential overrun condition),the tracker moves the start-of-symbol pointer backward toward the Aregion. When the pointer is moved into the overrun extend region (i.e.,A[8:4]) and a NULL symbol is detected, then the tracker moves thepointer forward 9 bits, which effectively removes the NULL symbol. Theremoval of the NULL symbol compensates for the overrun. One skilled inthe art will appreciate that the size of the overrun and the underrunextend regions can be adjusted based on the maximum possible differencesin clock frequencies between the transmitter and receiver and theguarantee frequency at which a NULL primitive is to be transmitted.

[0182]FIGS. 47A and 47B are diagrams illustrating NULL detection whilethe start-of-symbol pointer is not within the overrun or underrun extendregions in one embodiment. In this example, the pointer is at B[4]. Only8 valid bits are provided to the frame aligner (i.e., an underruncondition); as a result, the pointer is moved forward to B[4]. A NULLsymbol is also detected starting at B[4]. However, since the pointer isnot within the overrun or underrun extend regions, no NULL symbol isremoved or inserted and the two symbols of the NULL primitive areextracted.

[0183]FIGS. 48A and 48B are diagrams illustrating NULL detection whilethe start-of-symbol pointer is in the underrun extend region. Thepointer enters the underrun extend region as a result of thetransmitter's clock frequency being slightly slower than the receiver'sclock frequency. When the transmitter's clock frequency is slower, thereceiver can receive more data than the transmitter transmits and thereceiver will eventually run out of data as the start-of-symbol pointeris moved forward. When the pointer is in the underrun region, thetracker inserts a NULL symbol when another NULL symbol is detected toaccount for the underrun. In this example, the pointer is at B[2] when aNULL symbol is detected. The tracker moves the pointer backward to A[2](i.e., 1 symbol position) and inserts a NULL symbol at A[2]:B[3]. Theinserted NULL symbol and the two NULL symbols of the NULL primitive arethen extracted. In one embodiment, the inserted NULL symbol is aduplicate of the detected NULL symbol.

[0184]FIGS. 49A and 49B are diagrams illustrating NULL detection, whilethe start-of-symbol pointer is in the overrun extend region. The pointerenters the overrun extend region as a result of the transmitter's clockfrequency being slightly faster than the receiver's clock frequency.When the transmitter's local clock frequency is faster, the receivercannot receive as much data as the transmitter transmits, and so thereceiver would eventually lose some data. When the pointer is in theoverrun region, the tracker removes a NULL symbol of a NULL primitivewhen a NULL primitive is detected to account for the overrun. In thisexample, the pointer is at A[4] when a NULL symbol is detected. Thetracker moves the pointer forward 1 symbol position, which effectivelyremoves or skips over the NULL symbol at A[4]:B[5]. As a result, onlythe second NULL symbol of the NULL primitive is extracted. Allprimitives have the characteristic that each symbol of the primitiveencodes the same information. Thus, one of the symbols of a primitivecan be removed without loss of information.

[0185] From the foregoing, it will be appreciated that although specificembodiments of the invention have been described herein for purposes ofillustration, various modifications may be made without deviating fromthe spirit and scope of the invention. Accordingly, the invention is notlimited, except as by the appended claims.

I/we claim:
 1. A method in a receiving device for compensating for differences in clock frequencies between a transmitting device and the receiving device, the method comprising: receiving a stream of symbols transmitted serially from the transmitting device; identifying variable length groups of bits from the received stream along with an indication of whether the number of bits in the group represents an overrun or an underrun of bits; tracking whether an overrun or underrun criterion is satisfied based on the identified variable length groups of bits; and upon detection of an out-of-band symbol, removing the out-of-band symbol from the stream of symbols when the overrun criterion is satisfied; and inserting an additional out-of-band symbol into the stream when the underrun criterion is satisfied.
 2. The method of claim 1 wherein the tracking includes transferring the identified group of bits to a first-in-first-out bit buffer and maintaining a start-of-symbol pointer to indicate a starting bit of a symbol within the bit buffer.
 3. The method of claim 2 wherein the tracking includes upon detecting a synchronization symbol setting the start-of-symbol pointer to point to the start of the synchronization symbol.
 4. The method of claim 2 wherein the tracking includes moving the start-of-symbol pointer to an earlier received bit when an overrun of bits is indicated and moving the start-of-symbol pointer to later received bit when an underrun of bits is indicated.
 5. The method of claim 4 wherein the underrun criterion is satisfied when the start-of-symbol pointer is moved past a certain bit position within the bit buffer.
 6. The method of claim 4 wherein the overrun criterion is satisfied when the start-of-symbol pointer is moved before a certain bit position with the bit buffer.
 7. The method of claim 4 wherein the removing of an out-of-band symbol includes moving the start-of-symbol pointer to after the detected out-of-band symbol.
 8. The method of claim 4 wherein the inserting of an out-of-band symbol includes moving the start-of-symbol pointer a number of bit positions before the current start-of-symbol pointer and adding an out-of-band symbol at the start-of-symbol pointer, the number of bit positions moved being equal to the number of bits in a symbol.
 9. The method of claim 2 including extracting a symbol starting at the bit pointed to by the start-of-symbol pointer.
 10. The method of claim 1 wherein the detected out-of-band symbol is part of a primitive.
 11. The method of claim 10 wherein the inserting inserts of copy of the detected out-of-band symbol.
 12. The method of claim 10 wherein the primitive includes two out-of-band symbols.
 13. The method of claim 12 wherein the inserting includes inserting a copy of the first of the two out-of-band symbols.
 14. The method of claim 12 wherein the removing includes removing the first of the two out-of-band symbols.
 15. The method of claim 12 wherein the primitive is a control primitive.
 16. The method of claim 12 wherein the primitive is a synchronization primitive.
 17. A method in a receiving device for maintaining synchronization between the receiving device with a transmitting device, the method comprising: receiving variable length groups of bits from a received stream of bits of symbols along with an indication of whether the number of bits in the group represents an overrun or an underrun of bits; tracking whether an overrun or underrun criterion is satisfied based on the received variable length groups of bits; and upon receiving a symbol of a certain type, adjusting synchronization based on whether the overrun or underrun condition is satisfied.
 18. The method of claim 17 wherein the certain type of symbol is an out-of-band symbol.
 19. The method of claim 17 wherein the adjusting of the synchronization includes removing the received symbol of the certain type from the stream when the overrun criterion is satisfied and inserting an additional symbol of the certain type into the stream when the underrun criterion is satisfied.
 20. The method of claim 17 wherein the tracking includes transferring the received group of bits to a first-in-first-out bit buffer and maintaining a start-of-symbol pointer to indicate a starting bit of a symbol within the bit buffer.
 21. The method of claim 20 wherein the tracking includes upon detecting a synchronization symbol setting the start-of-symbol pointer to point to the start of the synchronization symbol.
 22. The method of claim 20 wherein the tracking includes moving the start-of-symbol pointer to an earlier received bit when an overrun of bits is indicated and moving the start-of-symbol pointer to later received bit when an underrun of bits is indicated.
 23. The method of claim 22 wherein the underrun criterion is satisfied when the start-of-symbol pointer is moved past a certain bit position within the bit buffer.
 24. The method of claim 22 wherein the overrun criterion is satisfied when the start-of-symbol pointer is moved before a certain bit position with the bit buffer.
 25. The method of claim 22 wherein the adjusting includes removing of the symbol of the certain type by moving the start-of-symbol pointer to after the out-of-band symbol when an overrun condition is satisfied.
 26. The method of claim 22 wherein the adjusting includes inserting of a symbol by moving the start-of-symbol pointer a number of bit positions before the current start-of-symbol pointer and adding a symbol of the certain type at the start-of-symbol pointer, the number of bit positions moved being equal to the number of bits in a symbol, when an underrun condition is satisfied.
 27. An aligner in a receiving device that maintains symbol alignment between the receiving device with a transmitting device, comprising: a detecting component that detects whether the clock frequency of the transmitting device is faster or slower than the clock frequency of the receiving device based on variable length groups of bits received from the transmitting device as part of a stream of bits of symbols; and an adjusting component that adjusts alignment based on whether the clock frequency of the transmitting device is faster or slower than the clock frequency of the receiving device.
 28. The aligner of claim 27 wherein the number of bits in the variable length groups of bits indicates whether the clock frequency of the transmitting device is faster or slower than the clock frequency of the receiving device.
 29. The aligner of claim 27 wherein the adjusting component removes a received symbol of a certain type from the stream when the clock frequency of the transmitting device is faster that the clock frequency of the receiving device and inserts an additional symbol of a certain type into the stream when the clock frequency of the transmitting device is slower than the clock frequency of the receiving device.
 30. The aligner of claim 27 including a bit buffer containing recently received variable length groups of bits and wherein the adjusting component moves forward a start-of-symbol pointer into the bit buffer one bit position when the number of bits in the variable length group of bits indicates that the clock frequency of the transmitting device is slower than the clock frequency of the receiving device and moves backwards the start-of-symbol pointer into the bit buffer one bit position when the number of bits in the variable length group of bits indicates that the clock frequency of the transmitting device is faster than the clock frequency of the receiving device.
 31. The aligner of claim 30 wherein the adjusting component moves backward the start-of-symbol pointer one symbol position when the start-of-symbol pointer has moved forward past a certain bit position and moves forward one symbol position the start-of-symbol pointer when the start-of-symbol pointer has moved backward past a certain bit position.
 32. The aligner of claim 31 wherein the moving backward and moving forward of the start-of-symbol pointer one symbol position occurs upon receiving a symbol of a certain type.
 33. The aligner of claim 32 wherein the symbol of the certain type is an out-of-band symbol.
 34. The aligner of claim 32 wherein the symbol of the certain type is a symbol of a primitive.
 35. The aligner of claim 32 wherein the moving backward of the start-of-symbol pointer one symbol position includes inserting a symbol into the stream of symbols and the moving forward of the start-of-symbol pointer one symbol position removing a symbol from the stream of symbols.
 36. The aligner of claim 27 including a synchronization component that detects a synchronization symbol in the stream of symbols and sets a start-of-symbol pointer to point to the start of the synchronization symbol. 